Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
41,872 characters · 19 sections · 72 citation commands
Learning Causal Models from Conditional Moment Restrictions by Importance Weighting
Consider learning the causal relationship between airline ticket prices and demand. As one might expect, prices and demand rise and fall through the seasons, being affected by other events like vacation periods, which are called confounders and may or may not be observable. Due to confounders, naively inferring from this pattern that higher (lower) prices increase (decrease) demand would be incorrect, and potentially detrimental. Thus, controlling for confounding effects is essential. This issue frequently arises in practice, especially when learning causal (structural) relationships is essential to answer counterfactual questions regarding policy intervention and outcome Hansen2022.
One approach to deal with confounding effects (like in the airline example above) is the instrumental variable (IV) approach Wooldridge2002,Greene2003Econometric. In the IV approach, the conditional moment restriction is defined as such that the causal model satisfies the restriction of zero expected value given IVs, thus conditioning out the confounding effect. The simplest representation of this idea is the two-step least squares (2SLS) for linear models Wooldridge2002,Greene2003Econometric. However, given the complex nature of the causal effect and confounding effect (and their relation), assuming a linear relation can be too strong. Thus, in this paper, we focus on nonparametric IV (NPIV) regressions, allowing for much more flexible estimation Newey2003.
NPIV can be viewed as an instance of a more general framework of causal inference under conditional moment restrictions. In this light, the machinery for inference under conditional moment restrictions also applies to NPIV, as well as its shortcomings. One major issue with using the conditional moment restrictions for causal inference is that one must approximate the conditional expectation, which is often difficult to do (see, e.g., Newey1993, Donald2003 for parametric and Newey2003,Ai2003Efficient for nonparametric IVs using sieves). For instance, Lewbel2007 and Otsu2011 estimate the conditional expectation by local kernel density estimation. However, local kernel density estimation suffers under high dimensionality. For this problem, recent methods suggest the use of machine learning methods, such as neural networks Hartford2017.
In this paper, we propose transforming conditional moment restrictions into unconditional moment restrictions by importance weighting using the conditional density ratio, which is defined as the ratio of the conditional probability density, conditioned on the IVs, to the unconditional probability density. We show that the unconditional expectation of a random variable weighted by the conditional density ratio is equal to the conditional expectation. Further, we show that it is possible to estimate the conditional density ratio with the least-squares method with a neural network. Once the conditional density ratio is estimated, the usual method of moments, such as GMM, can be used straightforwardly.
The contribution of this paper is as follows: (i) we propose a novel approach to convert conditional moment restrictions to unconditional moment restrictions by importance weighting; (ii) using our proposed transformation, we develop methods for NPIV; (iii) we analyze the estimation error
Among various problems of learning causal relationships from conditional moment restrictions, we focus on the NPIV regression for ease of discussion. Note that our proposed method can be applied to more general settings, similar to Ai2003Efficient.
Suppose that the observations $\{(Y_i, X_i, Z_{i})\}^n_{i=1}$ are i.i.d., where $Y_i\in\mathcal{Y}\subseteq\mathbb{R}$ is an observable scalar random variable, $X_i\in\mathcal{X}\subseteq \mathbb{R}^{d_X}$ is a $d_X$ dimensional explanatory variable, $Z_i\in\mathcal{Z}\subseteq \mathbb{R}^{d_Z}$ is a $d_Z$ dimensional random variable, called an IV, and $\mathcal{X}$ and $\mathcal{Z}$ are compact with nonempty interior. We also assume that the probability densities of $(Y_i, X_i, Z_i)$, $(Y_i, X_i)$, $Z_i$ exist and denote them by $p(y, x, z)$, $p(y, x)$, and $p(z)$, respectively. Let us define the causal relationships between $Y_i$ and $X_i$ as
where $f^*:\mathcal{X}\to\mathcal{Y}$ is a structural function, $\varepsilon_i$ is the sub-Gaussian error term with mean zero. To learn $f^*$, suppose the IV $Z_i$ satisfies the following conditional moment restrictions:
Then, we also assume that under the conditional moment restriction, we can uniquely identify $f^*$. Our goal is to learn $f^*$ from the conditional moment restrictions in ((ref)). If $Z_i = X_i$, this problem boils down to the estimation of the conditional expectation (regression function) $\mathbb{E}[Y_i| X_i]$. However, when $\mathbb{E}[\varepsilon_i| X_i] \neq 0$, $\mathbb{E}[Y_i| X_i]$ is not equivalent to $f^*(X_i)$; that is, typical regression analysis, such as least squares, may not return the correct estimate of $f^*(X_i)$.
In this section, we briefly review causal inference under moment restrictions.
One of the basic cases of using the IV is when $f^*$ is a linear model $X^\top_i\theta^*$ with $d_X$ dimensional vector $\theta^*$ and the error term $\varepsilon_i$ is correlated with the explanatory variable $X_i$. In this case, the parameter $\theta^*$ of the linear model can be estimated if there are IVs of dimension $d_X$ or more that satisfy the unconditional moment restrictions, $\mathbb{E}[Z_i \varepsilon_i] = \bm 0_{d_Z}$, where $\bm 0_d$ is a $d$ dimensional zero vector. In the just-identified case ($d_X = d_Z$), we can estimate $\theta^*$ as $\hat{\theta}_\text{IV} = \left(\frac{1}{n}\sum^n_{i=1}Z_i X^\top_i\right)^{-1}\frac{1}{n}\sum^n_{i=1}Z_i Y_i$. In the over-identified case ($d_X \leq d_Z$), we can estimate it by the 2SLS. In the 2SLS, we first regress $X_i$ by $Z_i$; then using a $d_X$ dimensional function $\hat{g}(Z_i)$ obtained from the first stage regression, we estimate $\theta^*$ as $\hat{\theta}_{\mathrm{2SLS}} = (\frac{1}{n}\sum^n_{i=1}\hat{X}_i \hat{X}^\top_i)^{-1}\frac{1}{n}\sum^n_{i=1}\hat{X}_i Y_i$, where $\hat{X}_{d,i} = Z^\top_i(\frac{1}{n}\sum^n_{j=1}Z_j Z^\top_j)^{-1} \frac{1}{n}\sum^n_{j=1}Z_j X_{d,j}$ and $\hat{X}_i = (\hat{X}_{1,i}, \dots, \hat{X}_{d_X,i})^\top$.
More generally, we can formulate the estimation of the linear structural function by unconditional moment restrictions defined using the IV; that is, $\mathbb{E}[m(\theta^*; Y_i, X_i, Z_i)] = \bm 0_{d_m}$, where $\theta^*\in\Theta$ is a parameter representing the causal effect, $\Theta$ is the parameter space, and $m:\Theta\times \mathbb{R} \times \mathcal{X} \times \mathcal{Z}\to \mathbb{R}^{d_m}$ is a $d_m$ dimensional moment function. Then, the GMM estimator is defined as $\hat{\theta}_{\text{GMM}} = \operatorname*{arg\,min}_{\theta\in\Theta} \left(\frac{1}{n}\sum^n_{i=1}m(\theta; Y_i, X_i, Z_i)\right)^\top W_{d_m} \left(\frac{1}{n}\sum^n_{i=1}m(\theta; Y_i, X_i, Z_i)\right),$ where $W_{d_m}$ is a $d_m\times d_m$ weight matrix. If $W_{d_m}$ is chosen as $W^*_{d_m} \propto \mathbb{E}[m(\theta^*; Y_i, X_i, Z_i)^\top m(\theta^*; Y_i, X_i, Z_i)]$, the estimator $\hat{\theta}_{\text{GMM}}$ is efficient under the posited moment conditions. Note that the GMM includes the 2SLS as a special case where $\varepsilon_i$ has the same variance among $i\in\{1,\dots,n\}$ and the zero covariance.
Three methods have been proposed to obtain $W^*_{d_m}$; two-step GMM, iterative GMM Hayashi, and continuous updating GMM Hansen1996. These estimators have the same asymptotic distribution, but different non-asymptotic properties. In particular, CUE is known as a special case of the generalized empirical likelihood (GEL) estimator Owen1988,Smith1997, which plays an important role in estimation with many moment restrictions NeweySmith2004.
In linear structural functions, zero covariance between the IV and error term suffices for identification. However, in nonparametric structural functions, we require a stronger restriction: the error term has conditional expectation zero given the IVs. Then, we can characterize the solution of NPIV as that of an integral equation $K(z) = \mathbb{E}[Y_i| Z_i = z] = \mathbb{E}[f^*(X_i) | z]=\int f^*(x) dF(x| z)$ with $F$ denoting the conditional c.d.f of $x$ given $z$. The identification also results in the uniqueness of the solution.
Several estimators have been proposed. Newey2003 proposes a nonparametric analogue of the 2SLS for linear structural functions. They first define the linear-in-parameter model as a model to approximate the nonparametric structural function $f^*$, where they use a linear approximation with basis expansion, such as sieve (series) regression. Let us denote the approximation as $f^*(X_i) \approx \psi(X_i)^\top\theta^*$, where $\psi:\mathcal{X}\to\mathbb{R}^{d_\psi}$, $d_\psi \leq n$ is a vector-valued function consisting of outputs of basis functions. Then, they conduct the 2SLS as follows: (i) they define a linear-in-parameter model as an approximation to the nonparametric model of $Z_i$, and estimate $\mathbb{E}[\psi(X_i)| Z_i]$ using that model; (ii) they regress $Y_i$ by $\mathbb{E}[\psi(X_i)| Z_i]$ to estimate $\theta^*$. In contrast, Ai2003Efficient proposes a nonparametric analogue of the GMM. Ai2003Efficient also approximate the nonparametric structural function $f^*$ by a linear-in-parameter model. However, unlike Newey2003, Ai2003Efficient estimates $\mathbb{E}[(Y_i - f(X_i))| Z_i]$, instead of estimating $\mathbb{E}[\psi(X_i)| Z_i]$, by another linear-in-parameter model. Using the estimator of $\mathbb{E}[(Y_i - f(X_i))| Z_i]$, Ai2003Efficient transforms conditional moment restrictions into unconditional ones and apply the GMM to estimate $f^*$. In addition to these two typical methods, Darolles2011, Carrasco2007 propose their own methods for NPIV.
This is how the NPIV problem is typically cast into the framework of conditional moment restrictions, where a parameter representing the causal relationship satisfies $\mathbb{E}[m(\theta^*; Y_i, X_i, Z_i)| Z_i] = \bm 0_{d_m}$. As with NPIV, we often define the moment function as a function of the nonparametric function $f$ instead of the parameter $\theta$. We can estimate $f^*$ defined under conditional moment restrictions using variants of GMM or empirical likelihood, e.g., Ai2003Efficient, Diminguez2004, Otsu2011, Lewbel2007, by transforming the conditional moment restrictions to unconditional ones.
{\bf 2SLS with more flexible models.} Hartford2017 extends the two-stage approach by employing a neural network density estimator, and Singh2019kernelIV does it by conditional mean embedding in RKHS. For example, in Hartford2017, they first approximate $K(z)$ by estimating $F(x| z)$ with neural networks. Then, for $B$ samples $\{\tilde{X}_j\}^B_{j=1}$ generated from the estimator $\hat{F}(x| z)$, they train neural networks by minimizing the empirical risk $\frac{1}{n}\sum^n_{i=1}\left(Y_i - \frac{1}{B}\sum_{\tilde{X}_j \sim \hat{F}(X_j| Z_i)} f(\tilde{X}_j) \right)^2$.
In contrast, Xu2021learning first approximates $f(X_i)$ by neural networks and regard the last layer as $\psi(X_i)$ in the 2SLS of NPIV. Then, they predict $\psi(X_i)$ by another model $\varphi(Z_i)$. Because $\psi(X_i)$ is transitive in the learning process, they alternatively train the models $\varphi(Z_i)$ and $f(X_i)$.
{\bf Minimax approach.} There has also been a recent surge in interest in minimax approaches that reformulate conditional moment restrictions as a minimax optimization problem Bennett2019deepgmm,Bennett2020Variational,Muandet2020Dual,Chernozhukov2020Adversarial,Liao2020Provably. For this approach, Dikkala2020 shows a strong theoretical guarantee on the estimation error bound.
Suppose that $p(y, x) > 0$ for all $(y,x)\in\mathcal{Y}\times \mathcal{X}$. Note that we have already assumed the existence of $p(y, x| z)$ for all $(y,x,z)\in\mathcal{Y}\times \mathcal{X}\times \mathcal{Z}$ by assuming the existence of the conditional moment restrictions. Define the conditional density ratio function $r:\mathcal{Y}\times \mathcal{X}\times \mathcal{Z} \to (0, C)$ as
for $0< C < \infty$. We assume the existence of the conditional density ratio function.
Then, we transform conditional moment restrictions into unconditional moment restrictions as
Thus, if we know the conditional density ratio function $r^*(y, x| z)$, we can approximate $\mathbb{E}[ (Y_i - f(X_i)) | z ]$ for $z\in\mathcal{Z}$ by the following sample average:
Based on this property, we propose the following two-stage method: (i) estimate the conditional density ratio function $r^*$; (ii) approximate conditional moment restrictions by the sample average of $\big(Y_i - f(X_i) \big)$ using the estimate of the conditional density ratio function $r^*$. Since we do not know the true value of the conditional density ratio $r^*$ and cannot calculate the expected value, we consider replacing $r^*$ with an estimator $\hat{r}$ and approximating the expected value with the sample mean.
First, we consider estimating $r^*(y, x| z)$. While it is possible to estimate the probability density functions of the numerator and the denominator, individually, following Vapnik’s principle Vapnik1998, we should avoid solving more difficult intermediate problems. Sugiyama2012 summarizes various methods to estimate the density ratio directly. Inspired by least-squares importance fitting (LSIF) of Kanamori2009, we estimate the conditional density ratio by minimizing \[\tilde{r} = \operatorname*{arg\,min}_{r\in\tilde{\mathcal{R}}} \frac{1}{2}\mathbb{E}_{Y,X}\left[\mathbb{E}_Z\left[\big(r^*(Y_i, X_i| Z_j) - r(Y_i, X_i| Z_j)\big)^2\right]\right],\] where $\tilde{\mathcal{R}}$ denotes a set of measurable functions and for a function $g:\mathcal{Y}\times\mathcal{X}\times\mathcal{Z}\to\mathbb{R}$, $\mathbb{E}_{Y,X}\left[\mathbb{E}_Z\left[g(Y_i, X_i, Z_j)\right]\right]$ denotes $\int g(y, x, z) p(y, x)p(z)\mathrm{d}y\mathrm{d}x\mathrm{d}z$. It is easy to confirm that $\tilde{r} = r^*$ by taking the first derivative of the risk. Because this risk includes the unknown $r^*$, it may seem intractable objective function. However, we can obtain the risk that does not include $r^*$ as
Here, we used $\mathbb{E}_{Y,X}\left[\mathbb{E}_Z\left[r^*(Y_i, X_i| Z_j)r(Y_i, X_i| Z_j)\right]\right] = \mathbb{E}_{Y,X}\left[\mathbb{E}_Z\left[\frac{p(Y_i, X_i, Z_j)}{p(Y_i, X_i)p(Z_j)}r(Y_i, X_i| Z_j)\right]\right] = \mathbb{E}_{Y,X,Z}\left[r(Y_i, X_i| Z_i)\right]$. For some hypothesis class $\mathcal{R}$, by approximating the risk with its sample approximation, we estimate the conditional density ratio as
For the hypothesis class $\mathcal{R}$, we can use various models, such as linear-in-parameter models and neural networks. suzuki2008 also proposes a similar formulation based on maximum likelihood estimation with constraints, which is not easy to solve with neural networks.
If we know the conditional density ratio function $r^*$, we can obtain unconditional moment restrictions $\mathbb{E}_{Y, X}\left[\big(\rho(f; Y_i, X_i, Z_1, r^*)\ \rho(f; Y_i, X_i, Z_2, r^*)\ \cdots\ \rho(f; Y_i, X_i, Z_n, r^*)\big)^\top\right] = \bm 0_{n}$, where $\rho(f; Y_i, X_i, z, r^*) = \big(Y_i - f(X_i)\big) r^*(Y_i, X_i| z) $. By replacing $r^*$ and its expectation with its estimator and the sample average, we obtain the sample vector moment restrictions, $\frac{1}{n}\sum^n_{i=1}\big(\rho(f; Y_i, X_i, Z_1, \hat{r})\ \rho(f; Y_i, X_i, Z_2, \hat{r})\ \cdots\ \rho(f; Y_i, X_i, Z_n, \hat{r})\big)^\top$.
Once we obtain the sample average, we can apply various methods for learning $f^*$ from the unconditional moment restriction. For instance, for a hypothesis class $\mathcal{F}$, we estimate $f^*$ by minimizing
This objective function is closely related with the projected mean squared error (MSE) introduced in Dikkala2020, defined as
The objective function ((ref)) is a special case of the GMM and GEL. In the GMM, we estimate $f^*$ by minimizing $\frac{1}{n}\sum^n_{j=1}\left(\frac{1}{n}\sum^n_{i=1} (Y_i - f(X_i)) \hat{r}(Y_i, X_i| Z_j)\right)^\top w_{j} \left(\frac{1}{n}\sum^n_{i=1} (Y_i - f(X_i)) \hat{r}(Y_i, X_i| Z_j)\right)$, where $w_j > 0$ is a weight Ai2003Efficient. Thus, our framework of transforming conditional moment restrictions to unconditional moment restrictions using importance weighting allows us to conduct causal inference using NPIV with conditional moment restrictions using a variety of models, such as linear-in-parameter models with basis functions and neural networks.
\paragraph{Linear-in-parameter models.} As an example, we introduce a linear-in-parameter model with some basis functions. Here, let us consider using the Gaussian kernel as the basis function. For $x \in \mathcal{\mathcal{X}}$, let $\varphi(x; \sigma^2)=\left(K(x, X_1; \sigma^2), \dots, K(x, X_n; \sigma^2)\right)^{\top}$, where $K(x, X_u; \sigma^2) = \exp\left(-\frac{\|x-X_u\|^2_2}{2\sigma^2}\right)$ is the Gaussian kernel with a hyperparameter $\sigma^2 > 0$ and $\|\cdot\|_2$ is the $L_2$ norm. Then, we define a linear-in-parameter model as $f(x; \sigma^2) = \beta^{\top}\varphi(x; \sigma^2)+\beta_{0}$, where $\beta\in\mathbb{R}^n$, and $\beta_{0}\in\mathbb{R}$.
\paragraph{Neural networks.} We can also use neural networks for approximating $f^*$. In this case, we need to carefully determine the network structures because we cannot uniquely determine the solution owing to the overparameterization. In existing studies, it is assumed that the network modes are well defined for the complexity of the nonparametric function $f^*$ and the sample size $n$.
\paragraph{Advantages of our proposed method.} Our method has the following three advantages: (i) our method is applicable to general causal inference problems, and is not limited to the NPIV problem formulated in Section (ref), as well as the Ai2003Efficient, (ii) our method can deal with high dimensional variables, owing to the machine learning technique. This is in contrast to related studies Ai2003Efficient and Otsu2011, which use a sieve and Nadaraya-Watson estimator, respectively, and hence do not work in high dimension settings, and (iii) our method is computationally efficient by the use of the importance weight approach. A similar study Hartford2017 requires additional sampling of $Y_i$ from an estimated conditional density $p(y|z,y)$. Another related study Dikkala2020 requires a difficult algorithm to solve its minimax optimization problem. Our importance weight approach avoids such computational burden.
In learning from moment restrictions, the structural function $f^*$ is determined by the equations that the expected value of random variables satisfy. When approximating the structure function $f^*$ using a model with more parameters than the sample size, the solution cannot be uniquely determined. When minimizing the prediction error directly, such overparameterization may not pose a major problem, and, in fact, may be necessary to improve the generalization error, as a recent finding suggests in the case of linear regression Bartlett2019. However, in NPIV, the objective function is the set of conditional moment restrictions. Unlike in direct prediction error minimization, a model trained to minimize empirically approximated conditional moment restrictions does not necessarily minimize the MSE for $f^*$ ($\mathbb{E}[(f(X_i) - f^*(X_i))^2]$). In fact, we empirically confirm that methods using neural network sometimes do not work well partly because of their overfitting to empirical moment minimization, not to the empirical MSE for $f^*$. Despite these potential problems, there is a strong motivation to use neural networks, owing to the reported superiority in some applications, such as computer vision Xu2021learning,xu2021deep,Yuan2021, natural language processing Ash2019,Chen2020mostly tasks. For this reason, we introduce a heuristic to avoid this problem.
As we explained, the parameters are not uniquely determined by conditional moment restrictions alone. Therefore, from the set of parameters satisfying conditional moment restrictions, we need to select a set of parameters that works well in prediction. We consider training a model that has the minimum MSE with $Y_i$ while satisfying conditional moment restrictions as follows:
Since it is difficult to solve constrained optimization with neural networks, we propose to solve the penalized optimization. In the case of linear combination with penalties, we train the model by
where $\eta \geq 0$ is a regularization coefficient.
The motivation of this heuristic is to select a function $f(x)$ that is the closest to $\mathbb{E}[Y_i| X_i=x]$, among multiple functions satisfying the conditional moment restriction. This heuristic works well when $f^*(x)$ takes a near value of $\mathbb{E}[Y_i| X_i=x]$ while $f^*(x)\neq \mathbb{E}[Y_i| X_i=x]$.
We show the estimation error of the conditional density ratio $r^*$ and structural function $f^*$. We denote the sample counterpart of $\mathcal{L}(f, r^*)$ as
Let us denote the distributions of $(Y_i, X_i)$ and $Z_i$ by $P$ and $Q$, respectively, and define the $L^2$ risk of a function $g$ with $P$ and $Q$ as $\|g\|^2_{L^2(P\times Q)} = \int \int g^2(w) \mathrm{d}P\mathrm{d}Q$. Let us denote the distribution of $(Y_i, X_i,Z_i)$ by $O$, and define the $L^2$ risk of a function $g$ with $O$ as $\|g\|^2_{L^2(O)} = \int g^2(w) \mathrm{d}O$. We put the following assumptions on the error term $\varepsilon_i$.
Note that the randomness of $Y_i$ depends on $\varepsilon_i$ and $X_i$. Define the hypothesis classes of the conditional density ratio $r^*$ and structural function $f^*$ as $\mathcal{R}$ and $\mathcal{F}$, respectively. Suppose that the hypothesis classes are Vapnik–Chervonenkis (VC) class (for rigorous definition, see Section 2.6 in van1996weak). This class include the true models, and the hypothesises are bounded.
We define a measure of complexity of the hypothesis classes: for $\mathcal{F}$, we define the complexity $\mathcal{C}_B(\mathcal{F}) := \int_0^B \sqrt{\log \mathcal{N}(\delta', \mathcal{F} ,\|\cdot\|_{L^\infty})} \mathrm{d}\delta'$ with a covering number $\mathcal{N}(\delta', \mathcal{F}, \|\cdot\|) := \inf\{N | \{f_j\}_{j=1}^N \mathrm{~s.t.~} \mathcal{F} \subset \cup_{j=1} \{f | \|f - f_j\| \leq \delta'\}\}$ in terms of a sup-norm $\|f\|_{L^\infty} = \sup_{x} |f(x)|$.
In this following parts, for the conditional density ratio estimation, we derive the bound of the MSE $\|\hat{r} - r^*\|_{L^2(P\times Q)}$; for the structural function estimation, we derive the bound of the projected MSE $\mathcal{L}(\hat{f}, {r}^*)$, which corresponds to the upper bound of the MSE of $\|\hat{f} - f^*\|_{L^2(O)}$ (Section (ref)).
First, we consider the MSE of the conditional density ratio. For a multilayer perception with ReLU activation function (Definition (ref)), we show the following bound. The proof is shown in Appendix (ref).
Next, we consider bounding the projected MSE of the structural function $f^*$. To bound the projected MSE, we use a technique associated with U-statistics. We first obtain the following lemma. Let $(Y_i, X_i)$ be $W_i$, and $W'_i$ be the i.i.d. copy of $W_i$. The proof is shown in Appendix (ref).
Regarding the form, we define an empirical version of $\widetilde{\mathcal{L}}(f,r)$ in an U-statistic form:
By a property of U-statistics (for example, see arcones1993limit), $\mathcal{L}_n(f,r) = \widetilde{\mathcal{L}}_n(f,r) + \mathcal{O}_{p}\left(1/n\right)$ clearly holds. Then, we can decompose the projected MSE $\mathcal{L}(\hat{f}, r^*)$ as follows:
We can handle $\Delta_r$ by the estimation error of $r^*$ as shown in Lemma (ref) in Appendix (ref). We evaluate the projected MSE by combining ((ref)) with Lemma (ref) and Lemma (ref):
This result reveals the following two findings: (i) the projected MSE is affected separately by the complexity of $\mathcal{F}$ and $\mathcal{R}$, and (ii) the overall convergence is $\mathcal{O}(1/\sqrt{n})$ when these complexities are finite. For instance, when using neural networks with Definition (ref), the assumptions on the hypothesis classes are satisfied (Lemma (ref)).
Following Chen2012 and Dikkala2020, we discuss the derivation of the upper bound of the MSE of $f^*$ from the projected MSE. Let us define the measure of ill-posedness Chen2012,Dikkala2020 with respect to the function class $\mathcal{F}$ as $\tau := \sup_{f\in\mathcal{R}} \frac{\|f- f^*\|_{L^2(O)}}{L(f, r^*)}$. Then, the MSE of $f^*$ is upper bounded as $\|f- f^*\|^2_{L^2(O)} \leq \tau^2 L(\hat{f}, r^*)$ Chen2012,Dikkala2020. Thus, Theorem (ref) also implies
Note that Dikkala2020 derived $1/n$ rate on the MSE of $f^*$.
We implement the following three methods based on our proposed method: first, we use neural networks to predict $f^*$ and train the model by penalized least-squares in ((ref)) (IW-LS); second, we use neural networks to predict $f^*$ and train the model by minimizing the sum of approximated moment restrictions in ((ref)) (IW-MM), which is the same as IW-LS except for the penalized term in the IW-LS; third, we use a linear-in-parameter model with the Gaussian kernel to predict $f^*$ and train the model by GMM (IW-Krnl). For all cases, we use neural networks for estimating $r^*$. We compare our proposed methods with four methods: DeepGMM (Bennett2019deepgmm), DFIV (Xu2021learning), DeepIV (Hartford2017), and KIV (Singh2019kernelIV). We use the datasets proposed in Newey2003, Ai2003Efficient, and Hartford2017. In addition, we train neural networks by simple least squares (LS), ignoring the dependency between $X_i$ and $\varepsilon_i$, as a comparison. To fairly evaluate the performances, for DeepGMM, DFIV, DeepIV, and KIV, we use the code and hyperparameters used in Xu2021learning\footnote{https://github.com/liyuan9988/DeepFeatureIV}. For IW-LS, IW-MM, IW-Krnl, and LS, we also follow the model and hyperparameters of the code as possible. More details are shown in Appendix (ref).
First, we investigate the performances of the proposed methods using econometric settings of Newey2003 and Ai2003Efficient. These settings have simpler structures than recently proposed settings, such as in Hartford2017. When using complex and high-dimensional datasets, there is an inherent difficulty in learning due to its complexity, separate from the problem setting. Therefore, we use simple datasets to check whether the proposed method can actually learn $f^*$.
Newey2003 generates $\{(Y_i, X_i, Z_i)\}^n_{i=1}$ as follows: first, they generate $\{(\varepsilon_i, U_i, Z_i)\}^n_{i=1}$ from the multivariate normal distribution $\mathcal{N}\left(
,\left(
\right)\right)$; then, they generate $X_i = Z_i + U_i$ and $Y_i=f^*(X_i)+\varepsilon_i$, where $f^*(X_i) = \ln (|X_i-1|+1) \operatorname{sgn}(X_i-1)$.
Ai2003Efficient generates $\{(Y_i, X_i, Z_i)\}^n_{i=1}$ as follows: first, they generate $\{(\varepsilon_i, X_{1i}, V_{i}, U_i)\}^n_{i=1}$ as $\varepsilon_i \sim \mathcal{N}\left(0, X_{1i}^{2}+V_{i}^{2}\right)$, $X_{1i} \overset{\mathrm{i.i.d.}}{\sim} \text {Unif}[0,1]$, $V_{i} \overset{\mathrm{i.i.d.}}{\sim} \text {Unif}[0,1]$, and $ U_i \overset{\mathrm{i.i.d.}}{\sim} \mathcal{N}\left(0, X_{1i}^{2}+V_{i}^{2}\right)$; second, they generate $ X_{2i}=X_{1i}+V_{i}+R \times \varepsilon_i+U_i$ and $Y_{1}=X_{1i}\gamma_{0}+h_{0}\left(X_{2i}\right)+\varepsilon_i$, where $h_{0}\left(X_{2i}\right)=\exp \left(X_{2i}\right)/\left(1+\exp \left(X_{2i}\right)\right)$ and $R$ is chosen as $0.9$; then, obtain $X_i = (X_{1i}\ X_{2i})^\top$ and $Z_i = (X_{1i}\ V_i)$. Here, $\varepsilon_i$ and $U_i$ are unobservable, and $f^*(X_i) = X_{1i}\gamma_{0}+h_{0}\left(X_{2i}\right)$, where the function $h_0$ and $\gamma_0$ are unknown.
We run each algorithm $20$ times on each dataset with $n = 1,000$ and calculate the mean squared error (MSE). In the left graph of Figure (ref) and Figure (ref), we report the MSEs. In the dataset of Newey2003, DFIV and our proposed IW-MM produce smaller MSE. The dataset only has a one-dimensional $X_i$, which mitigates the identification problem of neural networks. In contrast, in the dataset of Ai2003Efficient, IW-LS leads to smaller MSE. In Appendix (ref), using the experimental setting of Newey2003, we also show additional experimental results on the empirical convergence of the MSE of the IWMM and a comparison between the IWMM and classical 2SLS.
Because the dimensions of the IVs are low in the original settings, we add more IVs to the original settings and investigate the performances of the algorithms. The detailed settings are described in Appendix (ref). The results are shown in the right graph of Figures (ref) and Figure (ref).
To investigate the performance with more complicated and high-dimensional datasets, we use the demand design dataset for synthetic airline ticket sales proposed by Hartford2017. In this dataset, we observe $(Y_i, P_i, T_i, S_i, C_i)$, where $Y_i$ is sales, $P_i$ is price, $T_i$ is time, $S_i$ is consumer's emotion, and $C_i$ is cost to use as IV. Here, $T_i$ and $S_i$ are covariates, that is, $X_i = (T_i, S_i)$, and the IV is $Z_i = (C_i, T_i, S_i)$. The sales $Y_i$ is generated as $Y_i= 100+(10+P_i) S_i h(T_i)-2 P_i +\varepsilon_i$, where $h(t)=2\left(\frac{(t-5)^{4}}{600}+\exp \left(-4(t-5)^{2}\right)+\frac{t}{10}-2\right).$ Since $P_i$ is an endogenous variable and correlated with the IV, we generate $P_i$ to contain $C_i$ as $P_i=25+(C_i+3) h(T_i)+V_i$. In the simulation, we assume $\varepsilon_i \sim \mathcal{N}\left(\rho V_i, 1-\rho^{2}\right)$, $V_i \sim \mathcal{N}(0,1)$, $C_i \sim \mathcal{N}(0,1)$, $T_i$ is sampled from the uniform distribution with the continuous support $[0, 10]$, and $S_i$ is sampled from the uniform distribution with the discrete support $\{1,\dots, 7\}$. Here note that $\mathbb{E}[\varepsilon_i| C_i, T_i, S_i] = \mathbb{E}[\varepsilon_i| Z_i] = 0$. The extent of correlation between $P_i$ and $V_i$ is controlled by $\rho\in\{0.25,0.75\}$ and the larger the $\rho$ is, the more severe the correlation problem becomes.
Figure (ref) shows the results with $1,000$ samples. The results under other settings are reporetd in Appendix (ref). In this setting, IW-LS and LS outperform the other methods. We consider this is because $f^*(X_i)$ takes large values compared to the error term. Under this situation, a model trained to predict $Y_i$ may perform well because the influence of $\mathbb{E}[\varepsilon_i| X_i]\neq 0$ is limited. However, the purpose of using NPIV in the first place is because the influence is large, or else effects of $\mathbb{E}[\varepsilon_i| X_i] \neq 0$ can be ignored. Therefore, this dataset is often used in the existing studies, it may not be suitable.
This paper proposed a method for learning causal relationships from conditional moment restrictions. Our method is based on importance weighting using the conditional density ratio. The proposed method showed superior performance in experiments. We point out potential problems in recently proposed methods concerning identification and empirical performances.
The authors would like to thank Liyuan Xu for his constructive advice.