Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
57,745 characters · 13 sections · 87 citation commands
Kernel Conditional Moment Test via Maximum Moment Restriction
Many problems in causal inference, economics, and finance are often formulated as a conditional moment restriction (CMR): for correctly specified models, the conditional mean of certain functions of data is almost surely equal to zero Newey93:CMR,AI03:CMR. Rational expectation models---widely used in many fields of macroeconomics---specify how economic agents exploit available information to form their expectations in terms of conditional moments Muth61:Rational. Recent advances in causal machine learning also rely on the CMR including a generalized random forest (GRF) Athey19:GRF, orthogonal random forest (ORF) Oprescu19:ORF, double machine learning (DML) Chernozhukov18:DML, and nonparametric instrumental variable regression Bennett19:DeepGMM,Lewis18:AGMM among others; see also Hartford17:DIV,Singh19:KIV,Muandet19:DualIV and references therein.
Checking the validity of these moment restrictions is the first and foremost step to ensure that a model is correctly specified which constitutes a fundamental assumption for its estimation and inference. A model misspecification often creates biases to parameter estimates, inconsistency of standard errors, and invalid asymptotic distributions that hinder our subsequent inference based on the model. An overidentifying restriction test in the generalized method of moments (GMM) framework is one of the standard approaches to test a finite number of unconditional moment conditions Hansen82:GMM,Hall05:GMM. The $J$-test is an example of such tests Sargan58:OITest,Hansen82:GMM, and numerous tests have been developed in econometrics to deal with various sources of misspecification; see, e.g., Bierens17:Econometric for a review. This paper focuses on an important class of CMR-based specification tests known as the conditional moment (CM) tests Newey85:Specification,Tauchen85:CMT which have a long history in econometrics Hausman78:Specification,White81:Misspecification,Bierens17:Econometric.
Testing conditional moment restrictions becomes more challenging as an infinite number of equivalent unconditional moment restrictions (UMR) must be examined simultaneously (cf. Section (ref)). At first, Newey85:Specification and Tauchen85:CMT proposed to perform the overidentifying restriction test on a finite subset of the UMR. Unfortunately, the CM tests that rely only on a finite number of moment conditions cannot be consistent against all alternatives. Additional assumptions such as the global identification of selected moment conditions and sample-size dependent moment conditions are required to guarantee consistency Jong96:Dependence,Donald03:Likelihood. To overcome this limitation, Bierens82:ConsistentCM introduced the first consistent CM tests---known as integrated conditional moment (ICM) tests---by checking all moment conditions simultaneously Bierens97:Asymptotic. However, the ICM test depends on parametric weighting functions and nuisance parameters that limit its practical use. An alternative class of consistent CM tests, known as smooth tests, employ nonparametric kernel estimation Zheng96:NonparamTest, LiWang98:NonparamTest which also forms a basis for the generalized empirical likelihood approach Delgado06:Consistent,Tripathi03:CMRTest. However, they have non-trivial power only against local alternatives that approach the null at a slower rate than $1/\sqrt{n}$, and are susceptible to the curse of dimensionality (cf. Section (ref) for the discussion).
Inspired by a surge of kernel-based tests gretton2012kernel,Chwialkowski16:KGFT,Liu16:KSD, we propose to embed the CMR in a reproducing kernel Hilbert space (RKHS). By transforming CMR into a continuum of UMR in RKHS, the test statistic is defined as the maximum moment restriction (MMR) within the unit ball of the RKHS (cf. Section (ref)). We then show that the MMR corresponds to the RKHS norm of a Hilbert space embedding of conditional moments. Not only can the MMR capture all information about the original CMR, but it also has a closed-form expression that enables the practical ease of implementation (cf. Theorems (ref) and (ref)). The MMR allows us to develop a class of consistent CM tests that we call kernel conditional moment (KCM) tests (cf. Section (ref)). Furthermore, it considerably simplifies the parameter estimation problems based on the CMR. Our framework has relationships to existing methods in econometrics and machine learning (cf. Section (ref)). To the best of our knowledge, the Hilbert space embedding of conditional moment restrictions has not appeared elsewhere in the literature.\footnote{Carrasco00:Continuum and their follow-up work are the most relevant works from the econometric literature. We discuss this connection in Section (ref).}
All proofs can be found in Appendix (ref). The code of our experiments is available at \url{https://github.com/krikamol/kcm-test}.
We introduce the CMR in Section (ref) and then review the concepts of kernels and RKHS in Section (ref). Finally, we discuss the main assumptions in Section (ref).
Let $Z$ be a random variable taking values in $\mathcal{Z}\subseteq\mathbb{R}^p$ with distribution $P_Z$, $X$ a subvector of $Z$ taking values in $\ensuremath{\mathcal{X}}\subseteq\mathbb{R}^d$ with distribution $P_X$, and $\Theta\subset\mathbb{R}^r$ a parameter space. Following Newey93:CMR, we consider models where the only available information about the unknown parameter $\theta_0\in\Theta$ is a set of conditional moment restrictions
where $\bm{\psi}: \ensuremath{\mathcal{Z}}\times\Theta\to\mathbb{R}^q$ is a vector of generalized residual functions whose functional forms are known up to the parameter $\theta\in\Theta$. The expectation is always taken over all random variables that are not conditioned on. Note that there can be two different models that are observationally equivalent on the basis of (ref) alone although an ideal parameter $\theta_0$ is unique.
Several statistical problems can be formulated as (ref). In nonparametric regression models, $Z=(X,Y)$ where $Y\in\mathbb{R}$ is a dependent variable and $\bm{\psi}(Z;\theta) = Y-f(X;\theta)$. For conditional quantile models, $Z=(X,Y)$ and $\bm{\psi}(Z;\theta) = \mathbbm{1}\{Y < f(X;\theta)\}-\tau$ for the target quantile $\tau\in [0,1]$. In heterogeneous effect estimation, $Z=(X,T,Y)$ where $T$ is a vector of treatments and $\bm{\psi}(Z;\theta(X)) = (Y-\langle\theta(X),T\rangle)T$. For instrumental variable regression, $Z=(X,W,Y)$ where $W$ is an instrumental variable and $\bm{\psi}(Z;\theta) = (Y-f_\theta(X))$ and $\mathbb{E}[\bm{\psi}(Z;\theta)|W] = 0$ almost surely. When $Z$ admits the density $p(z;\theta)$, we can define the moment conditions in terms of the score function as $\bm{\psi}(Z;\theta) = \nabla_{\theta}\log p(Z;\theta)$ and use it for local maximum likelihood estimation.
\paragraph{Conditional moment tests.} Given an independent sample $(x_i,z_i)_{i=1}^n$ drawn from a distribution that satisfies the conditional moments (ref) and an estimate $\hat{\theta}$ of $\theta_0$, our goal is to perform specification testing: Given a function $\bm{\psi}$ and a parameter estimate $\hat{\theta}$, we test the null hypothesis
For instance, in the test of functional form of the nonlinear regression model Hausman78:Specification, the null hypothesis can be expressed as $H_0: \mathbb{E}[Y - f(X;\hat{\theta})|X] = 0$ where $\hat{\theta} = \arg\min_{\theta\in\Theta}\,\mathbb{E}[(Y - f(X;\theta))^2]$. In this case, $Z=(Y,X)$ and $\bm{\psi}(Z;\theta) = Y-f(X;\theta)$. This test allows us to detect misspecifications of the functional form of $f$.
In this work, we assume that $\hat{\theta}$ is obtained independently of the data that is used to test (ref). In many cases, however, $\hat{\theta}$ is estimated using this data and hence the test performance is also subject to the estimation error. A generalization of our framework to those cases will require more involved analyses, and we leave it to future work.
Let $\ensuremath{\mathcal{X}}$ be a non-empty set and $\ensuremath{\mathcal{F}}$ a Hilbert space consisting of functions on $\ensuremath{\mathcal{X}}$ with $\langle \cdot,\cdot\rangle_{\ensuremath{\mathcal{F}}}$ and $\|\cdot\|_{\ensuremath{\mathcal{F}}}$ being its inner product and norm, respectively. The Hilbert space $\ensuremath{\mathcal{F}}$ is called a reproducing kernel Hilbert space (RKHS) if there exists a symmetric function $k:\ensuremath{\mathcal{X}}\times\ensuremath{\mathcal{X}}\to\mathbb{R}$ called the reproducing kernel of $\ensuremath{\mathcal{F}}$ such that
The latter is called the reproducing property of $\ensuremath{\mathcal{F}}$. Every positive definite kernel $k$ uniquely determines the RKHS for which $k$ is a reproducing kernel aronszajn50reproducing.
Let $\{(\lambda_j,e_j)\}$ be pairs of positive eigenvalues and orthonormal eigenfunctions of $k$, i.e., $\int e_i(x)e_j(x) \, \mathrm{d} x = 1$ if $i=j$ and zero otherwise. By Mercer's theorem SteChr2008, the kernel $k$ has the spectral decomposition
where the convergence is absolute and uniform. As a result, for any $f\in\ensuremath{\mathcal{F}}$, we have $f(x) = \sum_j f_je_j(x)$ with $\sum_{j}f_j^2/\lambda_j < \infty$ where $f_j = \langle f,e_j\rangle_{\ensuremath{\mathcal{F}}}$, $\langle f,g\rangle_{\ensuremath{\mathcal{F}}} = \sum_jf_jg_j/\lambda_j$, and $\|f\|_{\ensuremath{\mathcal{F}}}^2 = \langle f,f \rangle_{\ensuremath{\mathcal{F}}} = \sum_j f_j^2/\lambda_j$.
Next, we introduce the notion of integrally strictly positive definite (ISPD) kernels and Bochner's characterization.
ISPD kernels are an important notion in kernel methods and are closely related to characteristic and universal kernels, see, e.g., Simon-Gabriel18:KDE.
The next result characterizes shift-invariant kernels $k(x,x') = \varphi(x-x')$ for some positive definite $\varphi$.
Examples of popular kernels are the Gaussian RBF kernel $k(x,x') = \exp(-\|x-x'\|_2^2/2\sigma^2), \sigma > 0$, Laplacian kernel $k(x,x') = \exp(-\|x-x'\|_1/\sigma), \sigma > 0$, and inverse multiquadric (IMQ) kernel $k(x,x') = (c^2 + \|x-x'\|_2^2)^{-\gamma}$, $c,\gamma > 0$. See, e.g., SteChr2008 for more examples.
Our subsequent analyses rely on these key assumptions.
Assumption (ref) ensures that all expectations of functions of $(X,Z)$ are independent of time. The regularity conditions (ref) are standard assumptions Hall05:GMM which ensure that $\bm{\psi}$ is well-defined, and hold in most models considered in the literature Hall05:GMM. By contrast, (ref) may not hold, especially in non-linear models. A local identifiability can be assumed instead by imposing additional constraints on $\Theta$. Testing whether the constraints are sufficient can then be done, for example, by examining the Jacobian at some parameter values Hall05:GMM. Lastly, (ref) implies that the RKHS $\ensuremath{\mathcal{F}}$ consists of bounded continuous functions SteChr2008 and is expressive enough (cf. Theorem (ref)).
This section presents the RKHS representation of the CMR. Let $\mathscr{F}$ be a set of measurable functions on $\ensuremath{\mathcal{X}}$. Then, $\mathbb{E}_{\mathit{XZ}}[\bm{\psi}(Z;\theta)f(X)] = \mathbb{E}_X[\mathbb{E}_{\mathit{Z}}[\bm{\psi}(Z;\theta)f(X)|X]] = \mathbb{E}_X[\mathscr{M}(X;\theta)f(X)]$ for any $f\in\mathscr{F}$ by the law of iterated expectation. That is, the CMR in (ref) implies an infinite set of unconditional moment restrictions
Equivalently, any $\theta_0\in\Theta$ that satisfies (ref) must also satisfy what we call a maximum moment restriction (MMR)
It is known that the implied moment restrictions (ref) and (ref) can be insufficient to globally identify the parameters of interest. We call $\mathscr{F}$ for which (ref) implies (ref) a sufficient class of instruments. In the context of this work, $\mathscr{F}$ must consist of infinitely many instruments for the CM test to be consistent against all alternatives. However, the $\sup$ operator also makes it hard to optimize (ref). We resolve these issues by choosing $\mathscr{F}$ to be a unit ball in a RKHS, which we show to be a sufficient class of instruments. As a result, (ref) can be solved analytically, the parameters of interest can be consistently estimated, and the resulting CM test is consistent against all fixed alternatives.
Recently, Lewis18:AGMM and Bennett19:DeepGMM also propose to estimate $\theta_0$ based on (ref) and $\mathscr{F}$ that is parameterized by deep neural networks. While they consider an estimation problem, we focus on hypothesis testing problems. Nevertheless, our formulation of CMR can also be used to estimate $\theta_0$ (cf. Section (ref) and Appendix (ref)). Note that the algorithms proposed in Lewis18:AGMM and Bennett19:DeepGMM require solving a minimax game, whereas our approach for estimation is simply a minimization problem.
To express (ref) using the RKHS, we first develop a representation of the CMR in a vector-valued RKHS of functions $f:\ensuremath{\mathcal{X}}\to\mathbb{R}^q$ Alvarez12:KVF. Let $\ensuremath{\mathcal{F}}$ be the RKHS of real-valued functions on $\ensuremath{\mathcal{X}}$ with reproducing kernel $k$ and $\ensuremath{\mathcal{F}}^q$ the product RKHS of functions $f := (f_1,\ldots,f_q)$ where $f_i\in\ensuremath{\mathcal{F}}$ for all $i$ with an inner product $\langle f,g \rangle_{\ensuremath{\mathcal{F}}^q} = \sum_{i=1}^q \langle f_i,g_i\rangle_{\ensuremath{\mathcal{F}}}$ and norm $\|f\|_{\ensuremath{\mathcal{F}}^q} = \sqrt{\sum_{i=1}^q\|f_i\|^2_{\ensuremath{\mathcal{F}}}}$. For $\theta\in\Theta$, we define an operator $M_\theta$ on $\ensuremath{\mathcal{F}}^q$ as
where $\psi_i$ denotes the $i$-th component of $\bm{\psi}$. This operator takes an instrument $f\in\ensuremath{\mathcal{F}}^q$ as input and returns the corresponding conditional moment restrictions.
The following lemma shows that $M_\theta$ satisfies the property of the original conditional moment restrictions.
Moreover, by Assumption (ref) and (ref), $|M_\theta f| \leq \sum_{i=1}^q\|f_i\|_{\ensuremath{\mathcal{F}}_i}\sqrt{\mathbb{E}[\psi_i(Z;\theta)\psi_i(Z';\theta)k(X,X')]} < \infty$ where $(X',Z')$ is an independent copy of $(X,Z)$. Hence, $M_\theta$ is a bounded linear operator. By Riesz's representation theorem, there exists a unique element $\bm{\mu}_{\theta}$ in $\ensuremath{\mathcal{F}}^q$ such that $M_\theta f = \langle f,\bm{\mu}_\theta\rangle_{\ensuremath{\mathcal{F}}^q}$ for all $f\in\ensuremath{\mathcal{F}}^q$. Indeed, by the reproducing property,
where $\bm{\xi}_{\theta}(x,z) := \left( \psi_1(z;\theta)k(x,\cdot),\ldots,\psi_q(z;\theta)k(x,\cdot) \right)$ is the feature map in $\ensuremath{\mathcal{F}}^q$ and $\xi^i_\theta$ denotes the $i$-th element of $\bm{\xi}_\theta$. The equalities above are well-defined since $\bm{\xi}_\theta(x,z)$ is Bochner integrable SteChr2008, i.e., $\mathbb{E}\|\bm{\xi}_\theta(X,Z)\|_{\ensuremath{\mathcal{F}}^p} \leq \sqrt{\mathbb{E}\|\bm{\xi}_\theta(X,Z)\|_{\ensuremath{\mathcal{F}}^p}^2} = \sqrt{\mathbb{E}[ \bm{\psi}(Z;\theta)^\top\bm{\psi}(Z;\theta)k(X,X)]} < \infty$.
In other words, $\bm{\mu}_\theta := \mathbb{E}[\bm{\xi}_\theta(X,Z)]$ is a representer of $M_\theta$ in $\ensuremath{\mathcal{F}}^q$. We define $\bm{\mu}_\theta$ as conditional moment embedding (CMME) of $\mathbb{E}[\bm{\psi}(Z;\theta)|X]$ in $\ensuremath{\mathcal{F}}^q$ relative to $P_X$.
4The CMME $\bm{\mu}_\theta$ takes the form of a kernel mean embedding of $P_{\mathit{XZ}}$ with $\bm{\xi}_\theta$ as the feature map Muandet17:KME. This is illustrated in Figure (ref). Hence, given an i.i.d. sample $(x_i,z_i)_{i=1}^n$ from $P_{\mathit{XZ}}$, we can estimate $\bm{\mu}_\theta$ simply by $\widehat{\bm{\mu}}_\theta := \frac{1}{n}\sum_{i=1}^n\bm{\xi}_\theta (x_i,z_i)$. The following theorem establishes the $\sqrt{n}$-consistency of this estimator.
Remarkably, $\widehat{\bm{\mu}}_{\theta}$ converges at a rate $O_p(n^{-1/2})$ that is independent of the dimension of $(X,Z)$ and the RKHS $\ensuremath{\mathcal{F}}^q$. This is an appealing property because estimation and inference based on $\widehat{\bm{\mu}}_\theta$ become less susceptible to the curse of dimensionality (see, e.g., Khosravi19:NPIA and references therein for the discussion). Under certain assumptions, TolstikhinSM17:Minimax established the minimax optimal rate for the kernel mean estimators like $\widehat{\bm{\mu}}_\theta$.
The next theorem shows that $\bm{\mu}_\theta$ provides a unique representation of the CMR $\ensuremath{\mathscr{M}}(X,\theta)$ in $\ensuremath{\mathcal{F}}^q$ relative to $P_X$.
To better understand Theorem (ref), consider when $q=1$ and $k(x,x')=\varphi(x-x')$ is a shift-invariant kernel. First, we have $\bm{\mu}_\theta(\cdot) = \mathbb{E}_X[\mathbb{E}_Z[\bm{\psi}(Z;\theta)k(X,\cdot)|X]] = \mathbb{E}_X[\mathbb{E}_Z[\bm{\psi}(Z;\theta)|X]k(X,\cdot)] = \mathbb{E}_X[\ensuremath{\mathscr{M}}(X;\theta)k(X,\cdot)]$. It is then easy to show using Theorem (ref) that $\bm{\mu}_\theta(\cdot) = \int_{\mathbb{R}^d} \phi(\omega;\theta) c(\omega,\cdot)\, \mathrm{d}\Lambda(\omega)$ where $c(\omega,y) = \exp(i\omega^\top y) \neq 0$ and $\phi(\omega;\theta) := \mathbb{E}_X[\ensuremath{\mathscr{M}}(X;\theta)\exp({i\omega^\top X})]$ is the Fourier transform (or characteristic function) of the Borel measurable function $\ensuremath{\mathscr{M}}(x;\theta)$ relative to $P_X$. Hence, if $\text{supp}(\Lambda)=\mathbb{R}^d$, the uniqueness of $\bm{\mu}_\theta$ follows from the uniqueness of $\phi(\omega;\theta)$. Bierens82:ConsistentCM was the first to observe the characterization of the CMR in terms of the integral transform and then used it to construct the consistent CM tests of functional form (cf. Section (ref)).
Theorem (ref) shows that $\bm{\mu}_{\theta}$ captures all information about $\mathbb{E}[\bm{\psi}(Z;\theta)|x]$ for every $x\in\ensuremath{\mathcal{X}}$ for which $P_X(x) > 0$. Consequently, estimation and inference on CMR can be performed by means of $\bm{\mu}_\theta$ using the existing kernel arsenal. As mentioned earlier, for each $f\in\ensuremath{\mathcal{F}}^q$ and $\theta\in\Theta$, the inner product $\langle f,\bm{\mu}_\theta\rangle_{\ensuremath{\mathcal{F}}^q} = \langle f,\mathbb{E}[\bm{\xi}_\theta(X,Z)]\rangle_{\ensuremath{\mathcal{F}}^q}$ can be interpreted as a restriction of conditional moments with respect to $f$. Moreover, the investigator can inspect $\bm{\mu}_{\theta}(x,z)$, which measures to what extent the moment conditions are violated at $(x,z)$, i.e., structural instability, in order to understand the nature of misspecification.
Based on the CMME $\bm{\mu}_\theta$, we can now define the MMR as
By Theorem (ref), $\mathbb{M}(\theta) \geq 0$ and $\mathbb{M}(\theta) = 0$ if and only if $\theta=\theta_0$. Put differently, $\mathbb{M}(\theta)$ measures how much the models associated with $\theta$ violate the original CMR in (ref).
To obtain an expression for $\mathbb{M}(\theta)$, we define a real-valued kernel $h_\theta:(\ensuremath{\mathcal{X}}\times\ensuremath{\mathcal{Z}})\times(\ensuremath{\mathcal{X}}\times\ensuremath{\mathcal{Z}})\to\mathbb{R}$ based on the feature map $\bm{\xi}_\theta:\ensuremath{\mathcal{X}}\times\ensuremath{\mathcal{Z}}\to\ensuremath{\mathcal{F}}^q$ as follows:
Then, a closed-form expression for $\mathbb{M}(\theta)$ in terms of the kernel $h_\theta$ follows straightforwardly.
Finally, Mercer's representation (ref) of $k$ allows us to interpret $h_\theta$ and $\mathbb{M}(\theta)$ in terms of a continuum of unconditional moment restrictions.
That is, we can interpret $\mathbb{E}[\bm{\zeta}_\theta^j(X,Z)]$ as the UMR with $e_j$ acting as an instrument. Moreover, $\mathbb{M}^2(\theta)$ can be viewed as a weighted sum of moment restrictions based on the sequence of weights and instruments $(\lambda_j,e_j)_j$. As a result, the CM test based on $\mathbb{M}^2(\theta)$ as a test statistic examines an infinite number of moment restrictions. Note that $(\lambda_j,e_j)_j$ are defined implicitly by the choice of $k$.
By virtue of Theorem (ref), we can reformulate the CM testing problem (ref) in terms of the MMR as
Given an i.i.d. sample $\{(x_i,z_i)\}_{i=1}^n$ from the distribution $P_{\mathit{XZ}}$, we consider the test statistic
which is in the form of $U$-statistics serfling1980approximation. Although there exist several potential estimators for $\mathbb{M}^2(\theta)$, we focus on (ref) as it is a minimum-variance unbiased estimator with appealing asymptotic properties. Moreover, (ref) also provides a basis for the estimation of $\theta_0$ simply by minimizing $\widehat{\mathbb{M}}_n^2(\theta)$ with respect to $\theta\in\Theta$. Preliminary results on estimation are given in Appendix (ref).
Next, we characterize the asymptotic distributions of $\widehat{\mathbb{M}}^2_n(\theta)$ under the null and alternative hypotheses.
As we can see, $n\widehat{\mathbb{M}}_n^2(\theta) < \infty$ with probability one under the null $\theta=\theta_0$ and diverts to infinity at a rate $\mathcal{O}(\sqrt{n})$ under any fixed alternative $\theta\neq \theta_0$. Hence, a consistent CM test can be constructed as follows: if $\gamma_{1-\alpha}$ is the $1-\alpha$ quantile of the CDF of $n\widehat{\mathbb{M}}^2_n(\theta)$ under the null $\theta=\theta_0$, we reject the null with significance level $\alpha$ if $n\widehat{\mathbb{M}}^2_n(\theta) \geq \gamma_{1-\alpha}$.
Unfortunately, the limiting distribution in (ref) and its $1-\alpha$ quantile do not have an analytic form. Following recent work on kernel-based tests Liu16:KSD,Chwialkowski16:KGFT,gretton2012kernel, we propose to approximate the critical values using the bootstrap method proposed by Arcones92:Bootstrap,Huskova93:BSConsist, which was previously used in Liu16:KSD. Specifically, we first draw multinomial random weights $(w_1,\ldots,w_n)\sim\text{Mult}(n; \frac{1}{n},\ldots,\frac{1}{n})$ and compute the bootstrap sample $\widehat{\mathbb{M}}_n^*(\theta) = (1/n^2)\sum_{1 \leq i\neq j \leq n}(w_i - 1)(w_j - 1)h_\theta((x_i,z_i),(x_j,z_j))$. We then calculate the empirical quantile $\hat{\gamma}_{1-\alpha}$ of $n\widehat{\mathbb{M}}^*_n(\theta)$. For degenerate $U$-statistics, $\hat{\gamma}_{1-\alpha}$ is a consistent estimate of $\gamma_{1-\alpha}$ Arcones92:Bootstrap,Huskova93:BSConsist.
We summarize our bootstrap kernel conditional moment (KCM) test in Algorithm 1. Note that the proposed test checks the CMR for a given parameter $\theta$ and does not take into account the estimation error of $\theta$. We defer a full treatment of interplay between parameter estimation and hypothesis testing to future work.
Existing CM tests can generally be categorized into two classes. The former is based on a transformation of CMR into a continuum of unconditional counterparts, e.g., Bierens82:ConsistentCM,Bierens90:FuncForm, Jong96:Dependence, Bierens97:Asymptotic, and Donald03:Likelihood to name a few. The latter employs nonparametric kernel estimation which includes Zheng96:NonparamTest,LiWang98:NonparamTest,Fan00:KernelICM among others. While both classes lead to consistent tests, they exhibit different asymptotic behaviors; see, e.g., Fan00:KernelICM,Delgado06:Consistent for detailed comparisons.
\paragraph{A continuum of unconditional moments.}
One of the classical approaches is to find a parametric weighting function $w(x,\eta)$ such that
for almost all $\eta\in\Xi\subseteq\mathbb{R}^m$ where $\eta$ is a nuisance parameter. Newey85:Specification and Tauchen85:CMT proposed the so-called M-test using a finite number of weighting functions. Since it imposes only a finite number of moment conditions, the test cannot be consistent against all possible alternatives and power against specific alternatives depends on the choice of these weighting functions. Jong96:Dependence and Donald03:Likelihood showed that this issue can be circumvented by allowing the number of moment conditions to grow with sample size. Although our KCM test generally relies on infinitely many moment conditions, one can impose finitely many conditions using the finite dimensional RKHS such as those endowed with linear and polynomial kernels or resorting to finite-dimensional kernel approximations.
Stinchcombe98:Nuisance showed that there exists a wide range of $w(x,\eta)$ that lead to consistent CM tests. They call these functions “totally revealing”. For instance, Bierens82:ConsistentCM proposed the first consistent specification test for nonlinear regression models using $w(x,\eta) = \exp(i\eta^\top x)$ for $\eta\in\mathbb{R}^d$. Similarly, Bierens90:FuncForm used $w(x,\eta) = \exp(\eta^\top x)$ for $\eta\in\mathbb{R}^d$. An indicator function $w(x,\eta) = \mathbbm{1}(\alpha^\top x \leq \beta)$ with $\eta=(\alpha,\beta) \in \mathbb{S}^d\times (-\infty,\infty)$ where $\mathbb{S}^d=\{\alpha\in\mathbb{R}^d : \|\alpha\|=1\}$ was used in Escanciano06:Projection and Delgado06:Consistent. Other popular weighting functions include power series, Fourier series, splines, and orthogonal polynomials, for example. In light of Theorem (ref), the KCM test falls into this category where weighting functions are eigenfunctions associated with the kernel $k$.
Since $w(x,\eta)$ depends on the nuisance parameter $\eta$, Bierens82:ConsistentCM suggested to integrate $\eta$ out, resulting in an integrated conditional moment (ICM) test statistic:
where $\Xi$ is a compact subset of $\mathbb{R}^d$, $\nu(\eta)$ is a probability measure on $\Xi$, and $\widehat{Z}_n(\eta) := (1/\sqrt{n})\sum_i\bm{\psi}(z_i;\theta)w(x_i,\eta)$. The limiting null distribution of the ICM test was proven to be a zero-mean Gaussian process Bierens90:FuncForm. Bierens97:Asymptotic also characterizes the asymptotic null distribution of a general class of real-valued weighting functions.
The following theorem establishes the connection between the KCM and ICM test statistics.
This theorem is quite insightful as it describes the KCM test statistic as the ICM test statistic $\widehat{T}_n(\theta)$ of Bierens82:ConsistentCM where the distribution on the nuisance parameter $\omega$ is a Fourier transform of the kernel. For instance, the Gaussian kernel $k(x,x')=\exp(-\|x-x'\|_2^2/2\sigma^2)$ corresponds to the Gaussian density $\Lambda(\omega) = \exp(-\sigma^2\|\omega\|_2^2/2)$; see Muandet17:KME for more examples. Note that both weighting functions and integrating measures are implicitly determined by the kernel $k$. Unlike ICM tests, KCM tests can be evaluated without solving the high-dimensional numerical integration (ref) explicitly. Moreover, KCM tests can be easily generalized to $\ensuremath{\mathcal{X}}$ that is not necessarily a subset of $\mathbb{R}^d$.
Carrasco00:Continuum also considers a similar setting that involves a continuum of moment conditions in RKHS. Their approach, however, differs significantly from ours. First, they consider a specific case where the Hilbert space is a set of square integrable functions of a scalar $t\in[0,T]$ with the unconditional moment conditions $\mathbb{E}[\bm{\psi}_t(X,\theta_0)] = \mathbf{0}$ for all $t\in [0,T]$. Second, their key question is to identify the optimal choice of weighting matrix in GMM. Third, estimation is actually based on a truncation of infinite moment conditions. Lastly, they also proposed the CM test similar to the ICM tests, but it can handle only the case with $Z\in\mathbb{R}$, while our test is applicable to any domain with a valid kernel.
\paragraph{Nonparametric kernel estimation.}
The second class of tests, known as smooth tests Zheng96:NonparamTest,LiWang98:NonparamTest,Fan00:KernelICM, adopts the statistic of the form
Based on the kernel estimator of $\mathbb{E}[\bm{\psi}(Z;\theta)|X]f(X)$, the empirical estimate of (ref) can be expressed as
where $K_{ij} = K((x_i - x_j)/h)$, $K(\cdot):\mathbb{R}^d\to\mathbb{R}$ is a normalized kernel function and $h$ is a smoothing parameter. Here, we emphasize that existing smooth tests rely on the kernel density estimator (KDE) in which the kernel used is not necessarily a reproducing kernel. For the smooth test to be consistent, $h$ must vanish as $n\to\infty$, whereas our KCM test is consistent even when the kernel is fixed. Nevertheless, if $K(\cdot)$ is a reproducing kernel, the test statistic $\widehat{T}_n(\theta)$ with a fixed smoothing parameter $h$ resembles the KCM test statistic (ref). In fact, Fan00:KernelICM has shown that the ICM test is a special case of the kernel-based test with a fixed smoothing parameter. However, the critical drawback of the nonparametric kernel-based tests is that they have non-trivial power only against local alternatives that approach the null at a slower rate than $1/\sqrt{n}$, due to the slower rate of convergence of kernel density estimators, i.e., $O((nh^{d/2})^{-1/2})$ as $h\to 0$ Fan00:KernelICM. Moreover, these tests are susceptible to the curse of dimensionality.
Last but not least, the kernel estimator is also a key ingredient in empirical likelihood-based CM tests Tripathi03:CMRTest,Kitamura04:EL,Dominguez04:GMM.
\paragraph{Kernelized Stein discrepancy (KSD).}
Stein's methods Stein72:Stein are among the most popular techniques in statistics and machine learning. One notable example is the Stein discrepancy which aims to characterize complex, high-dimensional distribution $p(x) = \tilde{p}(x)/N$ with intractable normalization constant $N = \int \tilde{p}(x)\, \mathrm{d} x$ using a Stein operator $\mathcal{A}_p$ such that
where $\mathcal{A}_p f(x) := \nabla_x\log p(x)f(x) + \nabla_{x} f(x)$. Here, we assume for simplicity that $x\in\mathbb{R}$. The Stein operator $\mathcal{A}_p$ depends on the density $p$ through its score function $s_p(x) := \nabla_x\log p(x) = \frac{\nabla_x p(x)}{p(x)}$, which is independent of $N$. When $p\neq q$, the expectation in (ref) gives rise to a discrepancy
See, also, Liu16:KSD. The Stein discrepancy has led to numerous applications such as variance reduction Oates17:Control and goodness-of-fit testing Liu16:KSD,Chwialkowski16:KGFT, among others.
Like (ref), we can observe that (ref) is indeed a set of unconditional moment conditions. To make an explicit connection between Stein discrepancy and CMR, we need to assume access to the probability densities. Let $\mathcal{P}_{\Theta}$ be a space of probability densities $p(z; \theta)$ such that $\theta \mapsto p(z;\theta)$ is injective. We choose $\bm{\psi}(z;\theta) = \nabla_z \log p(z;\theta) =: s_{\theta}(z)$ as the associated score function.\footnote{This differs from the standard definition of score function as $\nabla_\theta \log p(z|\theta)$ in the interpretation of maximum likelihood as generalized method of moments Hall05:GMM.} This yields the following CMR:
For any $\theta\in\Theta$, it follows that $\mathbb{E}[\bm{\psi}(Z;\theta)^\top f(X)] = \mathbb{E}[s_{\theta}(Z)^\top f(X) - s_{\theta_0}(Z)^\top f(X)] = \mathbb{E}[(s_{\theta}(Z) - s_{\theta_0}(Z))^\top f(X)] =: \Delta_f(\theta,\theta_0)$. While $\Delta_f(\theta,\theta_0)$ resembles the Stein discrepancy in (ref), we highlight the key differences. First, this characterization requires that the model is correctly specified, i.e., $p(z;\theta_0)$ is observationally indistinguishable from the underlying data distribution. Second, like the Stein discrepancy, it can be interpreted as the $f(x)$-weighted expectation of the score difference $s_{\theta} - s_{\theta_0}$. In contrast, the weighting function $f(x)$ in our setting depends only on $X$, which is a subvector of $Z$. We provide further discussion about this discrepancy measure in Appendix (ref). The following theorem follows directly from the preceeding observation.
Mostly related to our work are the RKHS-based Stein's methods Liu16:KSD,Chwialkowski16:KGFT. Specifically, if we assume the conditions of Theorem (ref) and that $f$ belongs to the RKHS, it follows that $\Delta(\theta,\theta_0) := \sup_f \|\Delta_f(\theta,\theta_0)\|_2$ coincides with the kernelized Stein discrepancy (KSD) proposed in Liu16:KSD and Chwialkowski16:KGFT. We will elaborate on this connection in further detail in future work.
We report the finite-sample performance of the KCM test against two well-known consistent CM tests, namely ICM test and smooth test, as discussed in Section (ref). We evaluate all tests with a bootstrap size $B=1000$ and a significance level $\alpha=0.05$.
\paragraph{Testing a regression function (REG).} We follow a similar simulation of regression model used in Lavergne16:Hausman. In this setting, for a given estimate $\hat{\bm\beta}$ of the regression parameters, the null hypothesis is $$H_0: \mathbb{E}[Y - \hat{\bm{\beta}}^\top X\,|\, X] = 0 \quad \text{a.s.}$$ where $X \in \mathbb{R}^d$ and $Y$ is a univariate random variable, i.e., $Z=(Y,X)$. The data are generated from the data generating process (DGP):
We set $\bm{\beta}_0 = \mathbf{1}$, and $X\sim\mathcal{N}(0,I_d)$. For the error term $e$, we consider two scenarios:
In each trial, we obtain an estimate of $\bm{\beta}_0$ by $\hat{\bm{\beta}} = \bm{\beta}_0 + \gamma$ where $\gamma \sim \mathcal{N}(\mathbf{0},\delta^2 I_{d})$. In this experiment, we set $d=5$. When $\delta=0$, the CMR are fulfilled, whereas they are violated, i.e., $H_0$ is false, if $\delta \neq 0$. Different values of $\delta$ correspond to different degrees of deviation from the null.
\paragraph{Testing the simultaneous equation model (SIMEQ).}
Following Newey90:Instrumental and Delgado06:Consistent, we consider the equilibrium model
where $Q$ and $P$ denote quantity and price, respectively, $R$ and $W$ are exogeneous variables, and $U$ and $V$ are the error terms. In this setting, $Z = (Q,P,R,W)$ and $X = (R,W)$. The null hypothesis can be expressed as
a.s. for some $\theta_0 = (\alpha_d,\beta_d,\alpha_s,\beta_s)$. We generate data according to $Q = \lambda_{11} R + \lambda_{12}W + V_1$ and $P = \lambda_{21} R + \lambda_{22}W + V_{2}$ where $R$ and $W$ are independent standard Gaussian random variables while $V_1$ and $V_2$ are correlated standard Gaussian random variables with $10^{-3}$ variance and $10^{-3}/\sqrt{2}$ covariance, and independent of $(R,W)$. We set $(\lambda_{11},\lambda_{12},\lambda_{21},\lambda_{22}) = (1,-1,1,1)$ and provide the details on how to find the true parameters $\theta_0$ in Appendix (ref). The estimate $\hat{\theta}$ is obtained as in the previous experiment. The null hypothesis corresponds to $\delta=0$ and different values of $\delta$ corresponds to alternative hypotheses. Rejecting $H_0$ means that the functional form of the supply and demand curves are misspecified.
Figure (ref) depicts the empirical results for $n\in \{1,2,4,6,8,10\}\times 10^2$ and $\delta\in\{10^{-4},2\times 10^{-3},4\times 10^{-3},6\times 10^{-3},8\times 10^{-3},10^{-2}\}$. First, it can be observed that KCM, ICM, and smooth tests are all capable of detecting the misspecification as the sample size and $\delta$ are sufficiently large. Second, the KCM test tends to outperform both ICM and smooth tests in terms of the test power, especially in a low sample regime (see Figure (ref)--(ref)) and a small deviation regime (see Figure (ref)--(ref)). In addition, the smooth test and the ICM test are competitive: there is no substantial evidence to conclude that one is always better than the other. Lastly, Figure (ref) in Appendix (ref) depicts that the Type-I errors of all tests are correctly controlled at $\alpha = 0.05$.
Lastly, we point out that this work does not elaborate on the effect of parameter estimation. In practice, the candidate parameter $\hat{\theta}$ has to be estimated from the observed data, which changes the asymptotic distribution of the test statistic. We envision the interplay between parameter estimation and hypothesis testing as an important arena for future work.
To conclude, we propose a new conditional moment test called the KCM test whose statistic is based on a novel representation of the conditional moment restrictions in a reproducing kernel Hilbert space. This representation captures all necessary information about the original conditional moment restrictions. Hence, the resulting test is consistent against all fixed alternatives, is easy to use in practice, and also has connections to existing tests in the literature. It also has an encouraging finite-sample performance compared to those tests. While the conditional moment restrictions have a long history in econometrics and so does the concept of reproducing kernel Hilbert spaces in machine learning, the intersection of these concepts remains unexplored. We believe that this work gives rise to a new and promising framework for conditional moment restrictions which constitute numerous applications in econometrics, causal inference, and machine learning.
We thank the anonymous reviewers for the helpful comments on our initial draft. KM is indebted to Motonobu Kanagawa and Junhyung Park for fruitful discussion.