Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
42,127 characters · 13 sections · 34 citation commands
Deep Generalized Method of Moments for Instrumental Variable Analysis
\afterpage{
}
Unlike standard supervised learning that models correlations, causal inference seeks to predict the effect of counterfactual interventions not seen in the data. For example, when wanting to estimate the effect of adherence to a prescription of $\beta$-blockers on the prevention of heart disease, supervised learning may overestimate the true effect because good adherence is also strongly correlated with health consciousness and therefore with good heart health cole2006drug. Figure (ref) shows a simple example of this type and demonstrates how a standard neural network (in blue) fails to correctly estimate the true treatment response curve (in orange) in a toy example. The issue is that standard supervised learning assumes that the residual in the response from the prediction of interest is independent of the features.
One approach to account for this is by adjusting for all confounding factors that cause the dependence, such as via matching rubin1973matching,kallus2016generalized or regression, potentially using neural networks shalit2017estimating,johansson2018learning,kallus2018deepmatch. However, this requires that we actually observe all confounders so that treatment is as-if random after conditioning on observables. This would mean that in the $\beta$-blocker example, we would need to perfectly measure all latent factors that determine both an individual's adherence decision and their general healthfulness which is often not possible in practice.
Instrumental variables (IVs) provide an alternative approach to causal-effect identification. If we can find a latent experiment in another variable (the instrument) that influences the treatment (\ie, is relevant) and does not directly affect the outcome (\ie, satisfies exclusion), then we can use this to infer causal effects angrist2008mostly. In the $\beta$-blocker example cole2006drug, the authors used co-pay cost as an IV. Because they enable analyzing natural experiments under mild assumptions, IVs have been one of the most widely used tools for empirical research in a variety of fields angrist2001instrumental. An important direction of research for IV analysis is to develop methods that can effectively handle complex causal relationships and complex variables like images that necessitate more flexible models like neural networks hartford2017deep,lewis2018adversarial.
In this paper, we tackle this through a new method called DeepGMM that builds upon the optimally-weighted Generalized Method of Moments (GMM) hansen1982large, a widely popular method in econometrics that uses the moment conditions implied by the IV model to {efficiently} estimate causal parameters. Leveraging a new variational reformulation of the efficient GMM with optimal weights, we develop a flexible framework, DeepGMM, for doing IV estimation with neural networks. In contrast to existing approaches, DeepGMM is suited for high-dimensional treatments $X$ and instruments $Z$, as well as for complex causal and interaction effects. DeepGMM is given by the solution to a smooth game between a prediction function and critic function. We prove that approximate equilibria provide consistent estimates of the true causal parameters. We find these equilibria using optimistic gradient descent algorithms for smooth game play daskalakis2017training, and give practical guidance on how to choose the parameters of our algorithm and do model validation. In our empirical evaluation, we demonstrate that DeepGMM's performance is on par or superior to a large number of existing approaches in standard benchmarks and continues to work in high-dimensional settings where other methods fail.
We assume that our data is generated by
where the residual $\epsilon$ has zero mean and finite variance, \ie, $\Eb{\epsilon} = 0$ and $\Eb{\epsilon^2}<\infty$. However, different to standard supervised learning, we allow for the residual $\epsilon$ and $X$ to be correlated, $\Eb{\epsilon\mid X}\neq0$, \ie, $X$ can be endogenous, and therefore $g_0(X)\neq\Eb{Y\mid X}$. We also assume that we have access to an instrument $Z$ satisfying
Moreover, $Z$ should be relevant, \ie $\Prb{X\mid Z} \neq \Prb{X}$. Our goal is to identify the causal response function $g_0(\cdot)$ from a parametrized family of functions $G=\braces{g(\cdot;\theta)\ :\ \theta\in\Theta}$. Examples are linear functions $g(x;\theta)=\theta^T\phi(x)$, neural networks where $\theta$ represent weights, and non-parametric classes with infinite-dimensional $\theta$. For convenience, let $\theta_0\in\Theta$ be such that $g_0(\cdot)=g(\cdot;\theta_0)$. Throughout, we measure the performance of an estimated response function $\hat{g}$ by its MSE against the true $g_0$.
Note that if we additionally have some exogenous context variables $L$, the standard way to model this using (ref) is to include them both in $X$ and in $Z$ as $X = (X',L)$ and $Z = (Z',L)$, where $X'$ is the endogenous variable and $Z'$ is an IV for it. In the $\beta$-blocker example, if we were interested in the heterogeneity of the effect of adherence over demographics, $X$ would include both adherence and demographics whereas $Z$ would include both co-payment and demographics.
{\bf Two-stage methods.} One strategy to identifying $g_0$ is based on noting that (ref) implies
If we let $g(x;\theta)=\theta^T\phi(x)$ this becomes $\Eb{Y\mid Z}=\theta_0^T\Eb{\phi(X)\mid Z}$. The two-stage least squares (2SLS) method angrist2008mostly first fits $\Eb{\phi(X)\mid Z}$ by least-squares regression of $\phi(X)$ on $Z$ (with $Z$ possibly transformed) and then estimates $\hat\theta^\text{2SLS}$ as the coefficient in the regression of $Y$ on $\Eb{\phi(X)\mid Z}$. This, however, fails when one does not know a sufficient basis $\phi(x)$ for $g(x,\theta_0)$. newey2003instrumental,darolles2011nonparametric propose non-parametric methods for expanding such a basis but such approaches are limited to low-dimensional settings. hartford2017deep instead propose DeepIV, which estimates the conditional density $\Prb{X=x\mid Z}$ by flexible neural-network-parametrized Gaussian mixtures. This may be limited in settings with high-dimensional $X$ and can suffer from the non-orthogonality of MLE under any misspecification, known as the “forbidden regression” issue angrist2008mostly (see (ref) for discussion).
{\bf Moment methods.} The generalized method of moments (GMM) instead leverages the moment conditions satisfied by $\theta_0$. Given functions $f_1,\dots,f_m$, (ref) implies $\Eb{f_j(Z)\epsilon} = 0$, giving us
A usual assumption when using GMM is that the $m$ moment conditions in (ref) are sufficient to uniquely pin down (identify) $\theta_0$.\todo[color=red!10]{Andrew: I added a footnote here discussing this given our comment in reviewer feedback that we would clarify. Not sure if this is over the top, and also not sure how else to clarify this assumption without just repeating it. Let me know what you think!}\footnote{This assumption that a finite number of moment conditions uniquely identifies $\theta$ is perhaps too strong when $\theta$ is very complex, and it easily gives statistically efficient methods for estimating $\theta$ if true. However assuming this is difficult to avoid in practice.} To estimate $\theta_0$, GMM considers these moments' empirical counterparts, $\psi_n(f;\theta)=\frac1n\sum_{i=1}^n{f(Z_i)(Y_i-g(X_i;\theta))}$, and seeks to make all of them small simultaneously, measured by their Euclidean norm $\fmagd v^2=v^Tv$:
Other vector norms are possible. lewis2018adversarial propose using $\fmagd v_\infty$ and solving the optimization with no-regret learning along with an intermittent jitter to moment conditions in a framework they call AGMM (see (ref) for discussion).
However, when there are many moments (many $f_j$), using any unweighted vector norm can lead to significant inefficiencies, as we may be wasting modeling resources to make less relevant or duplicate moment conditions small. The optimal combination of moment conditions, yielding minimal variance estimates is in fact given by weighting them by their inverse covariance, and it is sufficient to consistently estimate this covariance. In particular, a celebrated result hansen1982large shows that (with finitely-many moments), using the following norm in (ref) will yield minimal asymptotic variance (efficiency) for any consistent estimate $\tilde\theta$ of $\theta_0$:
Examples of this are the two-step, iterative, and continuously updating GMM estimators hansen1996finite. We generically refer to the GMM estimator given in (ref) using the norm given in (ref) as optimally-weighted GMM (OWGMM), or $\hat\theta^\text{OWGMM}$.
{\bf Failure of (OW)GMM with Many Moment Conditions.} When $g(x;\theta)$ is a flexible model such as a high-capacity neural network, many -- possibly infinitely many -- moment conditions may be needed to identify $\theta_0$. However, GMM and OWGMM algorithms fail when we use too many moment conditions. On the one hand, one-step GMM (\ie, (ref) with $\fmagd v=\fmagd v_p$, $p\in[1,\infty]$) is saddled with the inefficiency of trying to impossibly control many equally-weighted moments: at the extreme, if we let $f_1,\dots$ be all functions of $Z$ with unit square integral, one-step GMM is simply equivalent to the non-causal least-squares regression of $Y$ on $X$. We discuss this further in (ref). On the other hand, we also cannot hope to learn the optimal weighting: the matrix $C_{\tilde\theta}$ in (ref) will necessarily be singular and using its pseudoinverse would mean deleting all but $n$ moment conditions. Therefore, we cannot simply use infinite or even too many moment conditions in GMM or OWGMM.
We next present our approach. We start by motivating it using a new reformulation of OWGMM.
Let us start by reinterpreting OWGMM. Consider the vector space $\mathcal V$ of real-valued functions $f$ of $Z$ under the usual operations. Note that, for any $\theta$, $\psi_n(f;\theta)$ is a linear operator on $\mathcal V$ and $$ \mathcal C_{\theta}(f,h)=\frac1n\sum_{i=1}^n{f(Z_i)h(Z_i)(Y_i-g(X_i;\theta))^2} $$ is a bilinear form on $\mathcal V$. Now, given any subset $\mathcal F\subseteq\mathcal V$, consider the following objective function:
In other words, (ref) provides a variational formulation of the objective function of OWGMM and (ref) provides a saddle-point formulation of the OWGMM estimate.
In this section, we outline the details of our DeepGMM framework. Given our reformulation above in (ref), our approach is to simply replace the set $\mathcal F$ with a more flexible set of functions. Namely we let $\mathcal F=\{f(z;\tau):\tau\in\mathcal T\}$ be the class of all neural networks of a given architecture with varying weights $\tau$ (but not their span). Using a rich class of moment conditions allows us to learn correspondingly a rich $g_0$. We therefore similarly let $\mathcal G=\{g(x;\theta):\theta\in \Theta\}$ be the class of all neural networks of a given architecture with varying weights $\theta$.
Given these choices, we let $\hat\theta^\text{DeepGMM}$ be the minimizer in $\Theta$ of $\Psi_n(\theta;\mathcal F,\tilde\theta)$ for any, potentially data-driven, choice $\tilde\theta$. We discuss choosing $\tilde\theta$ in (ref). Since this is no longer closed form, we formulate our algorithm in terms of solving a smooth zero-sum game. Formally, our estimator is defined as:
Since evaluation is linear, for any $\tilde\theta$, the game's payoff function $U_{\tilde\theta}(\theta,\tau)$ is convex-concave in the functions $g(\cdot;\theta)$ and $f(\cdot;\tau)$, although it may not be convex-concave in $\theta$ and $\tau$ as is usually the case when we parametrize functions using neural networks. Solving (ref) can be done with any of a variety of smooth game playing algorithms; we discuss our choice in (ref). We note that AGMM lewis2018adversarial also formulates IV estimation as a smooth game objective, but without the last regularization term and with the adversary parametrized as a mixture over a finite fixed set of critic functions.\footnote{In their code they also include a jitter step where these critic functions are updated, however this step is heuristic and is not considered in their theoretical analysis.} In our experiments, we found the regularization term to be crucial for solving the game, and we found the use of a flexible neural network critic to be crucial with high-dimensional instruments.\todo[color=purple!10]{Tobias: Added a bit more discussion of AGMM here. Let me know what you think.}\todo[color=red!10]{Andrew: Updated this a bit further to reflect the additional difference that AGMM only parametrizes the critic as a mixture over a finite set of critic functions, which is also important.}
Notably, our approach has very few tuning parameters: only the models $\mathcal F$ and $\mathcal G$ (\ie, the neural network architectures) and whatever parameters the optimization method uses. In (ref) we discuss how to select these.
Finally, we highlight that unlike the case for OWGMM as in (ref), our choice of $\mathcal F$ is not a linear subspace of $\mathcal V$. Indeed, per (ref), replacing $\mathcal F$ with a high- or infinite-dimensional linear subspace simply corresponds to GMM with many or infinite moments, which fails as discussed in (ref) (in particular, we would generically have $\Psi_n(\theta;\mathcal F,\tilde\theta)=\infty$ unhelpfully). Similarly, enumerating many moment conditions as generated by, say, many neural networks $f$ and plugging these into GMM, whether one-step or optimally weighted, will fail for the same reasons. Instead, our approach is to leverage our variational reformulation in (ref) and replace the class of functions $\mathcal F$ with a rich (non-subspace) set in this new formulation, which is distinct from GMM and avoids these issues. In particular, as long as $\mathcal F$ has bounded complexity, even if its ambient dimension may be infinite, we can guarantee the consistency of our approach. Since the last layer in a network is a linear combination of the penultimate one, our choice of $\mathcal F$ can in some sense be thought of as a union over neural network weights of subspaces spanned by the penultimate layer of nodes.
Before discussing practical considerations in implementing DeepGMM, we first turn to the theoretical question of what consistency guarantees we can provide about our method if we were to approximately solve (ref). We phrase our results for generic bounded-complexity functional classes $\mathcal F,\mathcal G$; not necessarily neural networks.
Our main result depends on the following assumptions, which we discuss after stating the result.
(ref) proves that approximately solving (ref) (with eventually vanishing approximation error) guarantees the consistency of our method. We next discuss the assumptions we made.
(ref) stipulates that the moment conditions given by $\mathcal F$ are sufficient to identify $\theta_0$. Note that, by linearity, the moment conditions given by $\mathcal F$ are the same as those given by the subspace $\op{span}(\mathcal F)$ so we are actually successfully controlling many or infinite moment conditions, perhaps making the assumption defensible. If we do not assume (ref), the arguments in (ref) easily extend to showing instead that we approach some identified $\theta$ that satisfies all moment conditions. In particular this means that if we parametrize $f$ and $g$ via neural networks where we can permute the parameter vector $\theta$ and obtain an identical function, our result still holds. We formalize this by the following alternative assumption and lemma.\todo[color=red!10]{Andrew: Added formal lemma and proof justifying that consistency still holds under set identification as long as all identified theta give the same function g}
(ref) provides that $\mathcal F$ and $\mathcal G$, although potentially infinite and even of infinite ambient dimension, have limited complexity. Rademacher complexity is one way to measure function class complexity bartlett2002rademacher. Given a bound (envelope) as in (ref), this complexity can also be reduced to other combinatorial complexity measures such VC- or pseudo-dimension via chaining pollard1990empirical. bartlett2019nearly studied such combinatorial complexity measures of neural networks.
(ref) is needed to ensure that, for any $\theta$ with $\psi(f;\theta)>0$ for some $f$, there also exists an $f$ such that $\psi(f;\theta)>\frac14C_{\tilde\theta}(f,f)$. It trivially holds for neural networks by considering their last layer. (ref) similarly holds trivially and helps ensure that the moment conditions cannot simultaneously arbitrarily approach zero far from their true zero point at $\theta_0$. (ref) is a purely technical assumption that can likely be relaxed to require only nice (sub-Gaussian) tail behavior. Its latter two requirements can nonetheless be guaranteed by either bounding weights (equivalently, using weight decay) or applying a bounded activation at the output. We do not find doing this is necessary in practice.
{\bf Solving the Smooth Zero-Sum Game.} In order to solve (ref), we turn to the literature on solving smooth games, which has grown significantly with the recent surge of interest in generative adversarial networks (GANs). In our experiments we use the OAdam algorithm of daskalakis2017training. For our game objective, we found this algorithm to be more stable than standard alternating descent steps using SGD or Adam.
Using first-order iterative algorithms for solving (ref) enables us to effectively handle very large datasets. In particular, we implement DeepGMM using PyTorch, which efficiently provides gradients for use in our descent algorithms paszke2017automatic. As we see in (ref), this allows us to handle very large datasets with high-dimensional features and instruments where other methods fail.
{\bf Choosing $\tilde\theta$.} In (ref), we let $\tilde\theta$ be any potentially data-driven choice. Since the hope is that $\tilde\theta\approx\theta_0$, one possible choice is just the solution $\hat\theta^\text{DeepGMM}$ for another choice of $\tilde\theta$. We can recurse this many times over. In practice, to simulate many such iterations on $\tilde\theta$, we continually update $\tilde\theta$ as the previous $\theta$ iterate over steps of our game-playing algorithm. Note that $\tilde\theta$ is nonetheless treated as “constant” and does not enter into the gradient of $\theta$. That is, the second term of $U$ in (ref) has zero partial derivative in $\theta$.\todo[color=red!10]{Andrew: Clarification here added as promised in review response.} Given this approach we can interpret $\tilde\theta$ in the premise of (ref) as the final $\tilde\theta$ at convergence, since (ref) allows $\tilde\theta$ to be fully data-driven.
{\bf Hyperparameter Optimization.} The only parameters of our algorithm are the neural network architectures for $\mathcal F$ and $\mathcal G$ and the optimization algorithm parameters (\eg, learning rate). To tune these parameters, we suggest the following general approach. We form a validation surrogate $\hat \Psi_n$ for our variational objective in (ref) by taking instead averages on a validation data set and by replacing $\mathcal F$ with the pool of all iterates $f$ encountered in the learning algorithm for all hyperparameter choice. We then choose the parameters that maximize this validation surrogate $\hat \Psi_n$. We discuss this process in more detail in (ref).
{\bf Early Stopping.} We further suggest to use $\hat\Psi_n$ to facilitate early stopping for the learning algorithm. Specifically, we periodically evaluate our iterate $\theta$ using $\hat\Psi_n$ and return the best evaluated iterate.
In this section, we compare DeepGMM against a wide set of baselines for IV estimation. Our implementation of DeepGMM is publicly available at \url{https://github.com/CausalML/DeepGMM}.
We evaluate the various methods on two groups of scenarios: one where $X,Z$ are both low-dimensional and one where $X$, $Z$, or both are high-dimensional images. In the high-dimensional scenarios, we use a convolutional architecture in all methods that employ a neural network to accommodate the images. We evaluate performance of an estimated $\hat{g}$ by MSE against the true $g_0$.
More specifically, we use the following baselines:
Note that GMM+NN relies on being provided moment conditions. When $Z$ is low-dimensional, we follow AGMM lewis2018adversarial and expand $Z$ via RBF kernels around 10 centroids returned from a Gaussian Mixture model applied to the $Z$ data. When $Z$ is high-dimensional, we use the moment conditions given by each of its components.\footnote{That is, we use $f_i(Z) = Z_i$ for $i = 1,\ldots,\text{dim}(Z)$.}
{
}
In this first group of scenarios, we study the case when both the instrument as well as treatment is low-dimensional. Similar to lewis2018adversarial, we generated data via the following process:
In other words, only the first instrument has an effect on $X$, and $e$ is the confounder breaking independence of $X$ and the residual $Y-g_0(X)$. We keep this data generating process fixed, but vary the true response function $g_0$ between the following cases: $$ \text{\textbf{sin}: }g_0(x) = \sin(x)\qquad \text{\textbf{step}: }g_0(x)= \mathbbm{1}_{\{x \geq 0\}}\qquad \text{\textbf{abs}: }g_0(x)=\abs{x}\qquad \text{\textbf{linear}: }g_0(x)=x $$
We sample $n=2000$ points for train, validation, and test sets each. To avoid numerical issues, we standardize the observed $Y$ values by removing the mean and scaling to unit variance. Hyperparameters used for our method in these scenarios are described in (ref).\todo[color=red!10]{Andrew: Added appendix containing hyperparameter details.} We plot the results in (ref). The left column shows the sampled $Y$ plotted against $X$, with the true $g_0$ in orange. The other columns show in blue the estimated $\hat{g}$ using various methods. (ref) shows the corresponding MSE over the test set.
First we note that in each case there is sufficient confounding that the DirectNN regression fails badly and a method that can use the IV information to remove confounding is necessary.
Our next substantive observation is that our method performs competitively across scenarios, attaining the lowest MSE in each (except linear where are beat just slightly and only by methods that use a linear model). At the same time, other methods employing neural networks perform well in some scenarios and less well in others. Therefore we conclude that in the low dimensional setting, our method is able to adapt to the scenario and compete with best tuned methods for the scenario.
Overall, we also found that GMM+NN performed well (but not as well as our method). In some sense GMM+NN is a novel method; we are not aware of previous work using (OW)GMM to train a neural network. Whereas GMM+NN needs to be provided moment conditions, our method can be understood as improving further on this by learning the best moment condition over a large class using optimal weighting. AGMM performed similarly well to GMM+NN, which uses the same moment conditions. Aside from the heuristic jitter step implemented in the AGMM code, it is equivalent to one-step GMM, (ref), with $\fmagd\cdot_\infty$ vector norm in place of the standard $\fmagd\cdot_2$ norm. Its worse performance than our method perhaps also be explained by this change and by its lack of optimal weighting.\todo[color=red!10]{Andrew: I updated this paragraph to reflect fact that AGMM is not clearly worse than GMM+NN in Table 1}
In the experiments, the other NN-based method, DeepIV, was consistently outperformed by Poly2SLS across scenarios. This may be related to the computational difficulty of its two-stage procedure, or possibly due to sensitivity of the second stage to errors in the density fitting in the first stage. Notably this is despite the fact that the neural-network-parametrized Gaussian mixture model fit in the first stage is correctly specified, so DeepIV's poorer performance cannot be attributed to the infamous “forbidden regression” issue. Therefore we might expect that, in more complex scenarios where the first-stage is not well specified, DeepIV could be at even more of a disadvantage. In the next section, we also discuss its limitations with high-dimensional $X$.\todo[color=red!10]{Andrew: Updated DeepIV discussion to reflect the fact that first stage model is actually well-specified, so forbidden regression doesn't explain its poorer performance}
We now move on to scenarios based on the MNIST dataset lecun1998gradient in order to test our method's ability to deal with structured, high-dimensional $X$ and $Z$ variables. For this group of scenarios, we use same data generating process as in (ref) and fix the response function $g_0$ to be abs, but map $Z$, $X$, or both $X$ and $Z$ to MNIST images. Let the output of (ref) be $X^{\mathrm{low}}, Z^{\mathrm{low}}$ and $\pi(x)=\mathrm{round}(\min(\max(1.5x + 5, 0), 9))$ be a transformation function that maps inputs to an integer between 0 and 9, and let $\op{RandomImage}(d)$ be a function that selects a random MNIST image from the digit class $d$. The images are $28\times28=784$-dimensional. The scenarios are then given as:\todo[color=purple!10]{Tobias: I changed the notation here as the old own was awkwardly overriding variables}
We sampled 20000 points for the training, validation, and test sets and ran each method 10 times with different random seeds. Hyperparameters used for our method in these scenarios are described in (ref). We report the averaged MSEs in (ref). We failed to run the AGMM code on any of these scenarios, as it crashed and returned overflow errors. Similarly, the DeepIV code produced nan outcomes on any scenario with a high-dimensional $X$. Furthermore, because of the size of the examples, we were similarly not able to run Poly2SLS. Instead, we present Vanilla2SLS and Ridge2SLS, where the latter is Poly2SLS with fixed linear degree. Vanilla2SLS failed to produce reasonable numbers for high-dimensional $X$ because the first-stage regression is ill-posed.
Again, we found that our method performed competitively across scenarios, achieving the lowest MSE in each scenario. In the $\textbf{MNIST\textsubscript{Z}}$ setting, our method had better MSE than DeepIV. In the $\textbf{MNIST\textsubscript{X}}$ and $\textbf{MNIST\textsubscript{X,Z}}$ scenarios, it handily outperformed all other methods. Even if DeepIV had run on these scenarios, it would be at great disadvantage since it models the conditional distribution over images using a Gaussian mixture. This can perhaps be improved using richer conditional density models like chen2016infogan,isola2017image, but the forbidden regression issue remains nonetheless. Overall, these results highlights our method's ability to adapt not only to each low-dimensional scenario but also to high-dimensional scenarios, whether the features, instrument, or both are high-dimensional, where other methods break. Aside from our method's competitive performance, our algorithm was tractable and was able to run on these large-scale examples where other algorithms broke computationally.
Other related literature and future work. We believe that our approach can also benefit other applications where moment-based models and GMM is used hansen1980formulating,hansen1982generalized,berry1995automobile. Moreover, notice that while DeepGMM is related to GANs goodfellow2014generative, the adversarial game that we play is structurally quite different. In some senses, the linear part of our payoff function is similar to that of the Wasserstein GAN arjovsky2017wasserstein; therefore our optimization problem might benefit from a similar approaches to approximating the sup player as employed by WGANs. Another related line of work is in methods for learning conditional moment models, either in the context of IV regression or more generally, that are statistically efficient chamberlain1987asymptotic,ai2003efficient,blundell2007semi,chen2012estimation,chen2018optimal. This line of work is different in focus than ours; they focus on methods that are statistically efficient, whereas we focus on leveraging work on deep learning and smooth game optimization to deal with complex high-dimensional instruments and/or treatment. However an important direction for future work would be to investigate the possible efficiency of DeepGMM or efficient modifications thereof. Finally, there has been some prior work connecting GANs and GMM in the context of image generation ravuri2018learning, so another potential avenue of work would be to leverage some of the methodology developed there for our problem of IV regression. \todo[color=purple!10]{Tobias: Add other related work here.} \todo[color=red!10]{Andrew: Added additional requested references from Xiaohong and reviewers.}
Conclusions. We presented DeepGMM as a way to deal with IV analysis with high-dimensional variables and complex relationships. The method was based on a new variational reformulation of GMM with optimal weights with the aim of handling many moments and was formulated as the solution to a smooth zero-sum game. Our empirical experiments showed that the method is able to adapt to a variety of scenarios, competing with the best tuned method in low dimensional settings and performing well in high dimensional settings where even recent methods break.
This material is based upon work supported by the National Science Foundation under Grant No. 1846210.