Adversarial Estimators

Adversarial Estimators

\onehalfspacing

Introduction

Although it is not always obvious, nearly all population parameters that are estimated in econometrics and machine learning can be written as the solution of so-called saddle-point or adversarial objectives of the form:

where $l$ is a known loss function, $Y$ is a random variable and $\Theta, \Lambda$ are parameter spaces, containing the unknown parameter of interest $\theta_*$ and nuisance $\lambda$. We examine the natural estimator $\widehat\theta_n$ that approximately solves the empirical Nash condition:

which replaces the expectation $\mathbb{E}$ of the population objective (ref) with the average of $n$ iid samples, $\mathbb{E}_n$. We search for the estimators over so-called {\it sieve spaces} $\widehat\theta_n,\widehat\lambda_n\in\Theta_n,\Lambda_n$ (grenander1981abstract), which approximate the full parameter spaces $\Theta_n,\Lambda_n\subset\Theta,\Lambda$ and grow with the sample size $n$. These could be neural networks for example, growing in depth and width. The sequences $\widetilde \eta_n,\ \eta_n=o_\mathbb{P}(1)$ accommodate numerical procedures which only yield approximate Nash equilibria. This class of A-estimators (A for {\it adversarial}) strictly generalizes so-called M-estimators (M for {\it maximum likelihood-type}), which are obtained by fixing $\Lambda$ to be singleton.\\

A-Estimators have become a workhorse of econometrics and causal inference long before the advent of deep learning. CUEGMM's continuous-updating Generalized Methods of Moments (GMM), which looks for $\theta$ satisfying $\mathbb{E}[m(\theta,Y)]=0$ for some known function $m(\theta, Y)$, can be written as:

and is therefore an A-estimator, but not an M-estimator. In statistics, an earlier example consists of the Empirical Likelihood (EL) approach pioneered in cosslet1981, owen1988empirical, owen1990empirical, qin1994empirical. Subsequently, EL was unified with GMM into the Generalized Empirical Likelihood (GEL) framework (newey2004generalized), also subsuming the exponential-tilting estimator (imbens1995gel), for example. All GEL estimators are A-estimators, but their adversarial formulation was rarely salient. However, some of their benefits may be owed directly to their adversarial objective: the adversary $\lambda$ automatically detects which moment violations are most informative at a given parameter guess, adaptively guiding the estimation towards an efficient solution. This contrasts with earlier estimators which weighted the moments in a way that depended on choices of the researcher: the weights of Pearson's Method-of-Moments were manually set by the researcher (implicitly), resulting in inefficient root-$n$ asymptotics. Two-step GMM (hansen1982gmm) required choosing a first-step estimator to compute the weights, yielding inefficient higher-order asymptotics (see newey2004generalized). Formally, the optimal weights are nuisance parameters, and as we will see in Section (ref), estimating them via an adversary ensures that $\widehat{\theta}_n$ is robust to estimation errors in these nuisance parameters.\\

A key invention which put a spotlight on adversarial objectives in recent years were Generative Adversarial Networks, or GANs (goodfellow2014generative). They search for a generative model $Y\sim\mathbb{P}_\theta$ for which no adversary $\lambda(Y)\in(0,1)$ (called `critic' or `discriminator') could tell apart the generated data from $n$ real samples: $$\inf_{\mathbb{P}_\theta}\sup_{\lambda(\cdot)\in(0,1)} \mathbb{E}_{\mathbb{P}_\theta} \log\lambda(Y)+\mathbb{E}_n\log(1-\lambda(Y))$$ The objective contains the log-likelihood of a binary classifier $\lambda(Y)$ discriminating between an equal number of real and generated samples. As we show in Section (ref), this directly measures the Jensen-Shannon divergence between $\mathbb{P}_\theta$ and $\mathbb{P}_n$. As of today, versions of this objective are key to state-of-the-art image generation, see e.g. Jabbar2022ASO for a recent survey. An analogy to human-generated images makes this unsurprising: it is much easier to tell apart a photo from an image drawn by a human, than it is to draw a realistic image, or to define what makes a drawing realistic. This intuition motivates the objective: train the generator until its critic has nothing more to criticize. The ingenuity is that the researcher need not define a meaningful measure of `realism' of a piece of data anymore. Instead, this measure is learned by the adversary. It is clear that the utility of this idea extends beyond image generation: in Imitation Learning, a sub-field of Robotics, it has been used to teach human behavior to artificial agents without requiring hand-crafted measures of `humanness' (gail). In Econometrics, where new causal inference methods can usually only be benchmarked on simulated data sets, athey2019using used the objective to limit the impact of researcher's subjective choices by requiring simulations to be indistinguishable from real data. kaji2020adversarial proposed to use the objective to estimate structural economic models which produce realistic data beyond the set of features that would otherwise be manually specified by the researcher.\\

More generally, other adversarial objectives have proven useful beyond fitting models to data. In Reinforcement Learning, a sub-field of Robotics where agents independently discover strategies to reach predefined goals without copying prior examples, sbeed proposed an A-estimator in which the adversary detects and penalizes any systematic deviation from optimal behavior. constrainedOpt proposed an estimator which extends a standard ML objective by an adversary imposing fairness constraints across sub-populations. More recently, research in econometrics established A-estimators as a natural framework for integrating machine learning methods into causal inference, where quantities of interest are frequently identified by a continuum of restrictions. chernozhukov2020adversarial propose to estimate Riesz representers of causal parameters directly, via an adversary enforcing the restrictions identifying the Riesz representer. Estimating Riesz representers is key to obtaining well-behaved estimates of causal parameters in the presence of nuisance functions, and can also be useful for estimating asymptotic variances, e.g. penalizedsieve2019. Another line of research develops novel adversarial objectives to estimate causal parameters from {\it conditional} moment restrictions, which naturally arise from causal assumptions (e.g. the instrumental variable setting), and are usually more informative than any finite set of unconditional moment restrictions. In this line of research, the adversary can be viewed as adaptively finding the unconditional moment restriction which is most violated at the current parameter guess, among infinitely many which are implied by the conditional moment restriction. The key works are adversarialGMM2018,dikkala2020minimax and deepGMM2019, variationalMM2020. metzger2021 propose a semi-parametrically efficient generalization of GEL to the conditional case via adversarial networks, containing variationalMM2020 as a special case. \\

In summary, a recurring theme of adversarial objectives is that instead of manually defining which specific features of the data are important for a model to capture, the researcher's role is restricted to stating a general principle which should be satisfied by all features of the correct model, and the adversary {\it adaptively} focuses the estimation on the model's features which violate this principle the most. Over the course of the paper, we will encounter further interesting connections between various A-estimators, such as their Neyman orthogonality, their information-theoretic foundation via f-Divergences, and their ties to Lagrangian Duality.\\

Despite their popularity, we are not aware of a unified statistical theory of A-estimators. For some individual estimators, consistency (deepGMM2019) and convergence rate results (dikkala2020minimax, singh2018nonparametric,liang2018HowWell,belomestny2021rates) were obtained, but normality results are limited to parametric $\theta$, either in Kernel settings (variationalMM2020) or leaving high-level assumptions about neural networks unverified (kaji2020adversarial). This can be attributed to two main obstacles: the theory of M-estimation does not apply to A-estimators, and the arguments from which the former is built up are insufficient to e.g. obtain the required uniform convergence of the adversary. The second issue is that adversarial objectives are most popular in the context of (deep) neural networks, whose statistical analysis (particularly their asymptotic normality) is complicated, e.g. due to their non-convex sieve space. Even in M-estimation settings, it was not clear whether known, high-level conditions for normality could be verified for neural networks (cf. the Conclusion of shen2019asymptotic). We therefore make three separate contributions:

The remainder of the paper is structured as follows. In Section (ref), we review five different A-estimators proposed in the econometrics and machine learning literatures. We present our general statistical theory of A-estimators in Section (ref) and apply it in Section (ref) to derive novel results about the examples of Section (ref). We conclude by recapping the similar role adversaries play across all examples, providing intuition which types of problems may generally benefit from adversarial formulations. Appendix (ref) and Online Appendix (ref) contain the proofs omitted in Sections (ref) and (ref), respectively.\looseness=-1

Examples

Minimum $f$-Divergence

A powerful class of estimation objectives asymptotically minimize an $f$-divergence $D_f(\mathbb{P}_\theta \| \mathbb{P})$ between the distribution of the data $Y\sim \mathbb{P}=\mathbb{P}_{\theta_*}$ and the distribution of some model $\mathbb{P}_\theta, \theta\in\Theta_n$ with support $\mathcal{Y}$. This class, introduced by nowozin2016fgan, subsumes GANs (goodfellow2014generative), and many follow ups such as mao2017squares, chisqgan. For a continuous, proper convex function $f: \mathbb{R}\mapsto \mathbb{R}$ satisfying $f(1)=0$, the $f$-divergence is defined as $D_f(\mathbb{P}_\theta \| \mathbb{P})=\mathbb{E}_\mathbb{P}[f(\frac{\mathrm{d}\mathbb{P}_\theta(Y)}{\mathrm{d}\mathbb{P}(Y)})]$, where $\frac{\mathrm{d}\mathbb{P}_\theta(Y)}{\mathrm{d}\mathbb{P}(Y)}$ denotes the Radon-Nikodym derivative of $\mathbb{P}_\theta$ with respect to $\mathbb{P}$ (=likelihood ratio), which we assume exists for all $\theta\in\Theta$. Notably, $D_f(\mathbb{P}_\theta \| \mathbb{P})$ admits a useful dual representation:

where $f_*(t) := \sup_{\lambda\in\mathbb{R}} \lambda t-f(\lambda)$ denotes the {\it convex conjugate} of $f$. The equality above follows from $f=(f_*)_*$. Various choices\footnote{None of the objectives are unique: $f(t) \gets f(t)+c(t-1)$ for any $c$ yields the same divergence, but changes the expressions. Note that we may also swap $\mathbb{E}_{\mathbb{P}_\theta}$ and $\mathbb{E}_n$, which yields valid objectives for the respective “reverse” $f$-divergences.} for $f$ are presented in Table (ref). This duality is useful because the right-hand side suggests a finite-sample analog which does not depend on unknown quantities: we obtain an A-estimator for $\theta_*$ by letting

and solving for $\widehat\theta_n, \widehat\lambda_n$ satisfying the Nash condition (ref),(ref) in $\mathbb{E}_n[l(\theta, \lambda, Y)]$. Normalizing $f(t) \gets \frac{f(t)-f'(1)(t-1)}{f''(1)}$ without loss of generality\footnote{This implies $f'(1)=0$, $f''(1)=1$ and $f_*(0)=0, f_*'(0)=f_*''(0)=1$, which merely re-scales the divergence (ref) by a factor of $1/f''(1)$}, assuming the second derivative $f''$ exists, the function $\lambda$ attaining the supremum in (ref) at some $\theta$ is $\lambda_*^\theta=f'(\frac{\mathrm{d}\mathbb{P}_\theta}{\mathrm{d}\mathbb{P}})$. The adversary $\widehat\lambda_n$ therefore estimates this transformed likelihood ratio at the current guess for $\widehat\theta_n$, and the Nash-equilibrium corresponds to the case where it is approximately constant, i.e. the distribution $\mathbb{P}_{\widehat\theta_n}$ is close to that of the data. Notably, $\mathbb{E}_n[l(\theta, \lambda, Y)]$ can be evaluated using only samples from the two distributions\footnote{Note that we neither require explicit knowledge of $\mathbb{P}_{\theta}$ nor infinitely many samples from $\mathbb{P}_{\theta}$ at a given $n$: it suffices to draw $m \succ n^2$ Monte Carlo samples from $\mathbb{P}_{\theta}$ and solve for the corresponding finite sample saddle point. The resulting Monte Carlo approximation error for the expectation $\mathbb{E}_{\mathbb{P}_{\theta}}$ is then of order $\sqrt{m}^{-1}=n^{-1}$ and can thus be accounted for by letting $\widetilde\eta_n, \eta_n=O_\mathbb{P}(n^{-1})$ in equations (ref),(ref), which has no impact on our asymptotic results.}. This is crucial for GANs, where $\mathbb{P}_\theta$ is only implicitly defined via a push-forward mapping parametrized by a neural net. As proposed by kaji2020adversarial, this also makes it a drop-in alternative to the {\it Simulated Method of Moments}, which similarly estimates economic models from data they generate, but matches only a finite set of moments instead of the full distribution. \\

Generalized Empirical Likelihood

Our next example is a class of A-estimators that was proposed long before the recent success of adversarial objectives in deep learning. In econometrics, many important parameters $\theta_*$ are identified by a moment restriction of the form: $$\mathbb{E}[m(Y, \theta)]=0 \ \iff \ \theta=\theta_*$$ for some known, possibly vector-valued function $m(Y,\theta)$. In the Introduction, we presented the continuous-updating GMM objective (CUEGMM) for estimating $\theta_*$, a workhorse for causal inference in econometrics. In this section, we review the more general class of Generalized Empirical Likelihood (GEL) estimators (newey2004generalized), which solve the constrained minimization problem: $$\inf_{\Bar{\mathbb{P}},\theta\in\Theta} D_f(\Bar{\mathbb{P}}\|\mathbb{P}_n) \text{ s.t. } \mathbb{E}_{\Bar{\mathbb{P}}}[m(Y,\theta)]=0$$ That is, they seek for a parameter $\theta$ and a corresponding population distribution $\Bar{\mathbb{P}}$ that is as close as possible to the sample $\mathbb{P}_n$, subject to satisfying the moment constraint $\mathbb{E}_{\Bar{\mathbb{P}}}[m(Y,\theta)]=0$. At this high level, it is worth noting that GEL optimizes the same target as the objective in Section (ref), which imposes $\Bar{\mathbb{P}}=\mathbb{P}_\theta$ instead of a moment constraint. Glossing over some details, we can obtain a tractable estimator in this setting by concentrating out $\Bar{\mathbb{P}}$ from the corresponding Lagrangian:

Which again uses the convex conjugate $f_*$ of $f$ (see example (ref)). For a formal proof of this equivalence, see e.g. imbens1995gel. It is easy to see that GEL is an A-estimator with $l(\theta, \lambda , Y)=-f_*(\lambda'm(Y, \theta))$ where $\Lambda_n=\mathbb{R}^{\operatorname{dim}(m)}$ and $\Theta_n$ is the parameter space of the economic model. A particularly popular version of this objective corresponds to the case $D_f=\chi^2$, where Table (ref) tells us that $f(t)=(t-1)^2$ and $f_*(t)=t+t^2/4$. In this case, we can analytically solve for the optimal adversary given $\theta$. Substituting it in, we get the continuous-updating GMM objective presented in the introduction: $$\sup_{\lambda\in\mathbb{R}^{\operatorname{dim}(m)}} \mathbb{E}_n\left[-f_*\left(\lambda'm(Y, \theta)\right)\right] = \mathbb{E}_n\left[m(Y, \theta)\right]'\mathbb{E}_n\left[m(Y, \theta)m(Y, \theta)'\right]^{-1}\mathbb{E}_n\left[m(Y, \theta)\right]$$

Off-Policy Reinforcement Learning

Next, we review the Smoothed Bellman Error Embedding (SBEED) algorithm introduced by sbeed, a popular off-policy learning method in robotics. Off-policy learning aims to learn the optimal policy for an agent from data that was generated under an entirely different policy regime. This problem is not limited to robotics: since it was identified in the monetary policy context by LUCAS197619, it became a primary concern in econometrics and its recognition played a key role in the {\it credibility revolution} (credibilityrevolution) of econometrics. While problem definitions otherwise differ between these literatures, off-policy learning methods have received recent interest in econometrics (offpolicyathey, policylearning).\\

For an agent receiving reward $R(s,a)$ for taking action $a\in\mathcal{A}$ at state $s\in\mathcal{S}$, forming an expectation over the future state $s^{+}\in\mathcal{S}$, SBEED's goal is to learn the value function $V_*(s)$ and policy $a\sim P_*(\cdot|s)$ which satisfy the regularized Bellman equation: $$V_*(s)=\max_{P(\cdot|s)} \mathbb{E}_{a\sim P(\cdot|s)}\Big[R(s,a)+\beta \mathbb{E}_{s^+|s, a}[V_*(s^+)|s, a]\Big] + H(P, s)$$ where the entropy $H(P, s)=-\mathbb{E}_{a\sim P(\cdot|s)}[\log P(a|s)]$ regularizes the optimal policy $P_*(\cdot|s)$ towards exploring all actions $a\in\mathcal{A}$. Given the researcher's choice of $R, \beta$, the goal is to learn $P_*, V_*$ from finite samples $\{(s_i, a_i, s^+_i)\}_{i=1}^n$. Importantly, the actions $a_i$ may be sampled from a {\it suboptimal} policy which does not equal $P_*$. Starting from the first-order condition of the Bellman equation, sbeed develop an adversarial population objective, whose finite-sample analog is the A-estimator (ref),(ref) with loss:

where $\lambda(s, a)$, $\log P_\theta(a|s)$ and $V_\theta(s)$ are implemented as neural networks in practice.

A-Estimators for Conditional Moment Restrictions

Another powerful application for A-estimators recently pursued by the econometric literature are conditional moment estimators. These methods estimate parameters $\theta_*$ which are identified by restrictions of the form:

for some random variables $Y=(X,Z)$ and a known function $m(X,\theta)$. Conditions of this type occur e.g. when estimating some causal effect $\theta$ via instrumental variables, or as the first-order conditions of agents optimizing some expected utility given some information $Z$. As a result, nonparametric conditional moment estimators received considerable interest in econometrics, see e.g. aichen2003, aichen2007, chenqiu2016. These earlier estimators rely on first-step estimates of nuisance parameters capturing the conditional means and variances. Intuitively however, estimating the nuisance parameters via predictive objectives in a separate first step may dedicate scarce model capacity to capturing features which are not useful for the purpose of estimating $\theta_*$ downstream. This motivates recent work on adversarial objectives which unify the estimation into a single objective, more plausibly targeting the nuisance estimation towards the goal of identifying $\theta_*$. Specifically, we will examine the estimator of dikkala2020minimax, with $l(\theta,\lambda,Y)=m(X, \theta)'\lambda(Z)-\frac{1}{4}\|\lambda(Z)\|_2^2$, yielding the finite sample objective $$\inf_{\theta\in\Theta_n}\sup_{\lambda\in\Lambda_n} \mathbb{E}_n\left[ m(X,\theta)'\lambda(Z) - \frac{1}{4} \lambda(Z)'\lambda(Z) \right],$$ where $\Lambda_n$ is a class of neural networks. The methods proposed by Bennett2019DeepGM, variationalMM2020 are closely related, but differ in the penalty they impose on $\lambda$. dikkala2020minimax consider the case of instrumental variable regression, where $X=(y,x)$ and $m(X,\theta)=y-\theta(x)$, but we will examine the general case. We note that Example (ref) (SBEED, sbeed) can be viewed as a special case of re-scaled version of this objective, with $X=(s,a, s^+)$ and $Z=(s,a)$, although both literatures seem to be unaware of their connection. One can analytically solve for the optimal adversary $\lambda_*^\theta(Z)=2\mathbb{E}[m(X,\theta)|Z]$ to rewrite the population objective as: $$\mathbb{E}[l(\theta,Y)]:=\mathbb{E}[l(\theta, \lambda_*^\theta,Y)] =\mathbb{E}[\|\mathbb{E}[m(X,\theta)|Z]\|_2^2]=\mathbb{E}[\|\mathbb{E}[m(X,\theta)-m(X,\theta_*)|Z]\|_2^2]$$ which can be understood as a measure of distance between $\theta$ and $\theta_*$, which clearly attains its minimum at $\theta=\theta_*$, when $\mathbb{E}[m(X,\theta)|Z]\equiv0$. In Section (ref), we will apply our theory to derive the asymptotic distribution of this estimator and show that is in fact {\it inefficient}. We further discuss how the adversarial formulation of GMM can directly inform a simple modification similar to variationalMM2020 which yields an efficient A-estimator.

Estimating Riesz Representers

chernozhukov2020adversarial propose a distinct A-estimator to estimate {\it Riesz representers} for structural parameters $\phi_*$ which can be written as linear functionals $\phi_*=\phi(g_*)=\mathbb{E}[m(Y,g_*)]$. Here, $g_*=\mathbb{E}[y|x]$ is an unknown function for which an estimate $\widehat{g}_n$ is available from some first-stage regression of $y$ on $x$, where $Y=(y,x,w)$. Quantities like $\phi_*$ are common in the average treatment effect or asset pricing literature, for example. Unfortunately, especially if $\widehat{g}_n$ is estimated via machine learning, the `naive' estimator $$\widehat{\phi}_n=\mathbb{E}_n[m(Y,\widehat{g}_n)]$$ is often not well behaved: $\sqrt{n}(\widehat{\phi}_n-\phi_*)$ may not converge in distribution to a Gaussian limit and thus one cannot provide confidence intervals around the estimate. Under the conditions of the Riesz representation theorem however, there may exist a function $\theta_*\in\Theta$ called the {\it Riesz representer} of the functional $\phi(g)$, which satisfies: $$\phi(g)=\mathbb{E}[\theta_*(x)g(x)] \quad \forall g\in\Theta$$ If a well-behaved estimate $\widehat\theta_n$ of $\theta_*$ is available, it can be combined with $\widehat{g}_n$ to define the so-called {\it orthogonalized} estimator: $$\Tilde\phi_n = \mathbb{E}_n[m(Y,\widehat{g}_n)-\widehat\theta_n(x)(y-\widehat{g}_n(x))]$$ which attains asymptotic normality under rather weak conditions on $\widehat{g}_n$ (see Lemma 17 of chernozhukov2020adversarial). chernozhukov2020adversarial propose a generalized procedure to estimate $\widehat\theta_n$ via an A-estimator, which we will simplify as follows: $$\inf_{\theta\in\Theta_n}\sup_{\lambda\in\Lambda_n}\mathbb{E}_n[m(Y,\lambda)-\theta(x)\lambda(x)-\lambda(x)^2/2]$$ where $\Theta_n,\Lambda_n$ are neural networks. To clarify why this objective works, is it useful to analytically solve the adversarial component of the corresponding population objective: $$\sup_\lambda\mathbb{E}[m(Y,\lambda)-\theta(x)\lambda(x)-\lambda(x)^2/2]=\frac{1}{2}\mathbb{E}[(\theta_*(x)-\theta(x))^2]$$ As we will show in Section (ref), our theory directly yields the convergence rates for $\widehat\theta_n$ that chernozhukov2020adversarial's Lemma 17 requires for the asymptotic normality of $\Tilde\phi_n$. It does so at a reduced curse of dimensionality in $x$ for rather general function classes - i.e. under weaker conditions on smoothness and dimension of the data - complementing the original work.

General Theory

\paragraph{Roadmap.} This Section will present our general theory of A-estimators. Subsection (ref) briefly discusses an alternative definition of A-estimators that may be more natural to some readers. In Subsection (ref), we establish that A-estimators satisfy the desirable condition of Neyman-orthogonality with respect to the adversary and discuss its implications. Next, we characterize the convergence rates of A-estimators: Section (ref) provides a high-level result for arbitrary sieves such as splines or wavelets, not just neural nets. Under more easily verifiable low-level conditions, Subsection (ref) provides convergence rates for semiparametric settings involving neural networks, showing they exhibit a reduced curse-of-dimensionality. Finally, we characterize the asymptotic normality of smooth functionals of A-estimators. We again begin with a general, high-level result for arbitrary sieves, followed with the low-level conditions for the normality of neural networks. Notably, we show that a combination of undersmoothing and regularizing towards a convex target space suffices to overcome a key issue for normality proofs of neural networks: their non-convex sieve space.

\paragraph{Notation.} Throughout, we consider random variable $Y$ with support $\mathcal{Y}$, distribution $\mathbb{P}$ and corresponding expectation operator $\mathbb{E}$. We also denote the variance operator by $\mathbb{V}[f(Y)]=\mathbb{E}(f(Y)-\mathbb{E}[f(Y)])^2$ for any function $f:\mathcal{Y}\mapsto\mathbb{R}$. We denote the sample average, i.e. the expectation under the empirical distribution $\mathbb{P}_n$, by $\mathbb{E}_n$. Throughout, $\mathbb{E}$ will treat estimated parameters as deterministic sequences indexed by $n$, as is common in the literature. We also consider subvectors of $Y$, denoted by $x\in {\mathcal{X}}, \Bar x\in \Bar {\mathcal{X}}$, with their respective supports ${\mathcal{X}}, \Bar {\mathcal{X}}$ being subspaces of $\mathcal{Y}$. We require various norms: throughout, $\|x\|_q$ will denote the $\ell^q$ norm of a finite dimensional vector $x$, with $\|x\|=\|x\|_2$ being the Euclidean norm. For a possibly vector-valued function $f(x)$, we denote its $L^q$ function norm over some subset $\widetilde{\mathcal{X}}\subset\mathcal{X}$ by $\|f\|_{L^q(\widetilde{\mathcal{X}})}=\mathbb{E}[\|f(x)\|^q_q|x\in \widetilde{\mathcal{X}}]^{1/q}$. We denote the supremum norm of a vector $x$ with components $x_i$ by $\|x\|_\infty=\max_i |x_i|$. The supremum norm of $f$ over $\widetilde{\mathcal{X}}$ will be denoted by $\|f\|_{\widetilde{\mathcal{X}}}=\sup_{x\in {\widetilde{\mathcal{X}}}} \|f(x)\|_\infty$. For $\widetilde{\mathcal{X}}=\mathcal{X}$, we may omit the dependence on ${\mathcal{X}}$ by writing $\|f\|_\infty:=\|f\|_{\mathcal{X}}$. We will often write $a\prec b$ to denote $a=O(b)$, implying that a sufficiently large global constant $\infty>C>0$ exists such that $a\leq C b$, where $C$ does not depend on any varying aspects of the problem, such as any parameters, sample sizes, et cetera. We write $a\asymp b$ if $a\prec b \prec a$. We will also write $a\vee b = \max(a, b)$ and $a \wedge b = \min(a,b)$. Throughout, we will write $l^\theta(\lambda, Y)=l(\theta, \lambda, Y)$ and $l(\theta, Y)=l(\theta, \lambda_*^\theta, Y)$ for short, where $\lambda_*^\theta=\arg\max_{\lambda\in\Lambda} \mathbb{E} l(\theta, \lambda, Y)$. We denote by $\pi_n$ a (not necessarily linear) projection onto the respective sieves, i.e. $\pi_n \theta\in\arg\inf_{\theta'\in\Theta_n} \|\theta'-\theta\|_\infty$ for any $\theta\in\Theta$ and $\pi_n \lambda\in\arg\inf_{\lambda'\in\Lambda_n} \|\lambda'-\lambda\|_\infty$ for any $\lambda\in\Lambda$.

Nash vs Minimax

We presented our preferred definition for A-estimators in the introduction, as satisfying a Nash condition of the empirical loss. All results of this paper will apply to this definition. However, the reader may have noticed that the “simultaneous” Nash condition of the estimator is symmetric in $\widehat{\theta}_n$ and $\widehat{\lambda}_n$, unlike the `sequential' mini-max population objective, which nests a family of inner maximizations:

where the loss $l$ and as a result the solutions $\lambda_*^\theta$ are indexed by the parameter $\theta\in\Theta$. The reader may therefore wonder if we could define an A-estimator for $\theta_*$ in a similar `sequential' mini-max fashion. That is, we could consider a family of M-estimators $\widehat{\lambda}_{n}^\theta$ approximately maximizing the empirical loss at any value of $\theta\in\Theta$:

And then look for $\widehat{\theta}_n\in\Theta_n$ satisfying:

where $\Bar\eta_n=o_\mathbb{P}(1)$ again accommodates approximate minimization. Fortunately, it turns out that any $\widehat\theta_n$ satisfying the more compact Nash condition from the introduction always satisfies the mini-max condition presented above, as summarized by the following Lemma:

This reassures us that it suffices to find one set of values $\widehat\theta_n,\widehat\lambda_n$ which satisfy the Nash condition from the introduction, rather than a continuum of solutions $\widehat\lambda_n^\theta$ indexed by $\theta$. The final $\widehat\theta_n$ will satisfy the mini-max condition regardless, for some (unknown) $\widehat\lambda_n^\theta$. For our theory, it was crucial to derive the uniform convergence of $\widehat\lambda_n^\theta$, hence we will state the rate results for the more general mini-max definition. For the normality result, it was more convenient to work with the stronger Nash definition.

Adversaries are Neyman-Orthogonal

For many A-estimators, one could construct non-adversarial estimators which capture the same population objective. Whenever the adversarial nuisance parameter $\lambda$ is a function, this usually requires a non-parametric first-step estimation of an alternative nuisance parameter. However, such an alternative estimator may not have a desirable property that is guaranteed for A-estimators: {\it Neyman-orthogonality} of $\theta_*$ with respect to the nuisance parameter.\\

This property has a long history in statistics, dating back at least to neyman1959. It was popularized in econometrics by Chernozhukov2017DoubleDebiasedNeymanML as a key setting in which standard machine learning methods can be applied without invalidating causal inference, which sparked follow-up work such as Chernozhukov2021AutomaticDM seeking to reformulate non-orthogonal problems as orthogonal ones. The notion applies to parameters which are identified by a moment restriction of the form:

where $\varphi$ is known and $\nu_*$ is an unknown nuisance parameter which has to be estimated in a first step. A popular estimator $\widehat{\theta}_n$ in this setting would be hansen1982gmm's GMM, for example. The moment condition above is called (Neyman-)orthogonal whenever:

Intuitively, this states that the condition identifying $\theta_*$ is “locally robust” against perturbations in $\nu_*$. This guarantees that the uncertainty introduced by an appropriate first-step estimation of $\nu_*$ has no first-order effect on the GMM estimator $\widehat{\theta}_n$. Specifically, the asymptotic distribution of $\widehat{\theta}_n$ is the same as in the case in which $\nu_*$ is known. In contrast, when moment restrictions do not satisfy this orthogonality condition, uncertainty about $\nu_*$ generally amplifies the asymptotic variance of $\widehat{\theta}_n$, see e.g. chenliao2015, and normality may break down altogether.\\

Notably, if (and only if) $\theta_*$ is parametric, we can examine the first order condition for $\theta_*$ that is implied by the A-estimation objective (ref) in this moment restriction framework\footnote{Note however that even when $\theta_*$ is parametric, we usually cannot estimate it via GMM as $\nabla_\theta l(\theta, \widehat\lambda_n^\theta, Y)$ will not exist if $\widehat\lambda_n^\theta$ is a typical sieve, such as a neural network. For the same reason, the theory developed in this paper must not rely on any finite-sample first order conditions. Instead, it will use only the approximate Nash condition (ref), (ref).} : let $\varphi(\theta, \nu_*, Y)=\nabla_\theta l(\theta, \nu_*(\theta), Y)$, where $\nu_* : \Theta \mapsto \Lambda$ denotes the functional evaluating to $\nu_*(\theta)=\lambda_*^\theta$. Orthogonality then follows from the continuum of first-order conditions identifying $\nu_*$:

since the derivative operators are exchangeable. This implies that as $\widehat\theta_n$ approaches $\theta_*$, an A-estimator $\widehat\theta_n$ is robust to estimation errors in the adversary $\widehat\lambda_n^\theta$ relative to $\lambda_*^\theta$, meaning they do not reduce the accuracy of $\widehat\theta_n$, to a first-order.\\

Consider the example of Section (ref), which estimates $\theta_*$ minimizing the f-Divergence between the model $\mathbb{P}_\theta$ and the data $\mathbb{P}$. As a non-adversarial alternative, we could re-parametrize the problem and estimate $\nu_*:=d\mathbb{P}$ via a first-step Kernel density estimator $\widehat{\nu}_n(Y)=\widehat{d\mathbb{P}}_n(Y)$, and subsequently approximate the f-Divergence as the average over $f\left(\frac{d\mathbb{P}_\theta(Y)}{\widehat{\nu}_n(Y)}\right)$. However, the first-order condition for $\theta_*$ would not satisfy orthogonality, hence a GMM estimator based on this condition may not attain the variance of the analogous GMM estimator using $\nu_*$ instead. In contrast, the A-estimator of Section (ref) {\it does} attain this variance - due to its orthogonal adversary - which we formally establish in Section (ref). Moreover, this remains true when generalizing to a setting in which $\theta_*$ contains unknown functions, where no analogous GMM estimator exists that could capture the continuum of first-order conditions in $\theta_*$.\looseness=-1

Convergence Rate of A-Estimators

We begin with a general theorem characterizing the convergence rates of sieve A-estimators, for arbitrary loss functions and parameter spaces. It can be viewed as a generalization of shen1994's M-estimator result. Its proof is provided in Appendix (ref), with the main challenge being that shen1994's chaining arguments need to be carefully modified to hold uniformly over $\Theta$. Our theorem adopts a more compact formulation than shen1994 which does not require any norm over $\Theta,\Lambda$ to state our assumptions, although convergence rates are obtained for any (pseudo-)norm $d(\theta,\theta_*)$ which is dominated by the objective.

Semiparametric Rates with Neural Networks

Next, we will apply the general result of the previous section to derive the convergence rates for neural network A-estimators. For generality, we will consider the semiparametric setting in which $\theta,\lambda$ may contain both Euclidean vectors and functions. These lower-level conditions are easy to verify in practice, but are general enough to apply to all estimators considered in Section (ref). We will include the proof as it is short and an instructive application of Theorem (ref). The theorem allows for two types of function classes, both of which can be viewed as generalizations of traditional Hölder functions with $D$-dimensional domain, with their own notion of an {\it intrinsic dimension} $d^*\leq D$, which may be smaller than that of the data $D$. As we will review in Remark (ref), we observe that neural networks achieve a reduced curse of dimensionality in these settings.\looseness=-1

\onehalfspacing

Asymptotic Normality of A-Estimators

In applications, it we a often interested in estimating a quantity of the form $F(\theta_*)$, where $F:\Theta\mapsto\mathbb{R}$ is some known functional. To derive confidence intervals around the plug-in estimate $F(\widehat\theta_n)$, we need its asymptotic distribution. To this end, we present Theorem (ref), which can roughly be viewed as a generalization of shen1997onmethods to A-Estimators. For this section, we make use of the pathwise derivative presented in Definition (ref). We require a particular inner product over the space $\Theta$: \[\langle \theta, \theta' \rangle := \nabla_{\theta_*\to\theta} \nabla_{\theta_*\to\theta'} \mathbb{E}[l(\theta_*, Y)]\] As discussed in Definition (ref), the notation $\nabla_{\theta_*\to\theta}$ implicitly assumes that the corresponding limit exists and is linear in $\theta$. For short, we write $\lambda_*^{\prime\theta}[v]:=\nabla_{\theta\to v}\lambda_*^\theta$, $l'(\theta, Y)[v]:=\nabla_{\theta\to v}l(\theta, Y)$ and $l'(\theta,\lambda, Y)[v, w]:=\nabla_{\theta\to v} l(\theta,\lambda, Y) + \nabla_{\lambda\to w} l(\theta, \lambda, Y)$.

Semiparametric Normality with Neural Networks

Next, we present Theorem (ref), which strengthens the assumptions of our previous neural network convergence rate result (Theorem (ref)) in a way that allows us to derive the asymptotic normality of functionals $F(\widehat\theta_n)$ via Theorem (ref). A crucial innovation is that we are able to work around the non-convexity issues of deep neural networks discussed in Remark (ref), to obtain a normality result from {\it low-level} conditions, which only consist of general properties that the loss function must satisfy (A4-A7), and certain {\it implementation choices} for the neural networks that must be followed. To the best of our knowledge, the theorem therefore also provides the first low-level conditions for the normality of smooth functionals of deep neural network {\it M-estimators} (as the special case where $\Lambda$ is singleton).

Application to Examples

Minimum $f$-Divergence

Applying our general Theorem (ref) to the estimator of Section (ref) yields Proposition (ref), which provides the convergence rate of semiparametric $\widehat\theta_n$ if $\Lambda$ and all unknown functions in $\Theta$ are approximated by classes of neural networks.

While a fast convergence rate of the model distribution $\mathbb{P}_{\widehat\theta_n}$ is a key goal in semi- and nonparametric estimation, whenever some function $F(\widehat\theta_n)$ of the estimate informs downstream decision-making, we are often interested in obtaining confidence intervals around $F(\widehat\theta_n)$. To this end, we derive the asymptotic normality of the adversarial $f$-Divergence objective - an entirely novel result at this level of generality, to the best of our knowledge. First, we compute the inner product defined in Section (ref), which can be expressed concisely: $$ \langle \theta, \theta' \rangle = \nabla_{\theta_*\to\theta}\nabla_{\theta_*\to\theta'} \mathbb{E}\left[f\left(\frac{\mathrm{d}\mathbb{P}_{\theta_*}(Y)}{\mathrm{d}\mathbb{P}(Y)}\right)\right] = \mathbb{E}\left[\nabla_{\theta_*\to\theta}\log \mathrm{d}\mathbb{P}_{\theta_*}(Y) \cdot \nabla_{\theta_*\to\theta'}\log \mathrm{d}\mathbb{P}_{\theta_*}(Y)\right] $$ Where $\nabla_{\theta_*\to\theta}\log \mathrm{d}\mathbb{P}_{\theta_*}(Y)=\nabla_{\theta_*\to\theta}\frac{\mathrm{d}\mathbb{P}_{\theta_*}(Y)}{\mathrm{d}\mathbb{P}(Y)}$ is a pathwise derivative of the Radon-Nikodym derivative. Conditions under which the normality result of Section (ref) applies are presented in Proposition (ref).

Note that $\langle\cdot,\cdot\rangle$ and hence the asymptotics of $\widehat\theta_n$ are independent of $f$, so the $f$-divergences are asymptotically equivalent. An example for a smooth functional $F(\theta)$ that is of particular interest in the semiparametric setting $\theta=(\beta, \alpha)$ is $F(\theta)=\beta'\zeta$, which “picks out” a linear combination of the parametric components. This allows us to derive the asymptotic normality of the vector $\sqrt{n}(\widehat\beta-\beta_*)$ in the following Corollary, which makes use of the orthogonal scores assumption that is standard in the semiparametric literature.

The $f$-GAN objective therefore attains the efficient asymptotics of maximum likelihood, but does not require explicit knowledge of the model density $\mathbb{P}_\theta$.

Generalized Empirical Likelihood

For the class of Generalized Empirical Likelihood estimators introduced in Section (ref), the $\sqrt{n}$-normality and asymptotic efficiency of $\widehat{\theta}_n$ is long established in the parametric case (imbens1995gel, imbens2002GLE, newey2004generalized). However, our theoretical framework still allows us to extend the known results to the semiparametric case where $\theta$ may contain unknown functions, which are approximated by a class of neural networks $\Theta_n$ which may grow with $n$. In this case, we can characterize the convergence rate of $\widehat{\theta}_n$ to the identified set $\Theta_*=\{\theta\in\Theta:\mathbb{E}[m(Y, \theta)]=0\}$, which is unlikely to be singleton given that an infinite-dimensional parameter is hardly pinned down by a finite number of unconditional moment restrictions. We obtain the following result:

Off-Policy Reinforcement Learning

Next, we will use our theory to the extend the known results about SBEED, the off-policy RL algorithm of sbeed introduced in Section (ref). Theorem (ref) makes it easy to obtain the convergence rates of the corresponding A-estimator:

A-Estimators for Conditional Moment Restrictions

We will now apply our theory to examine the asymptotic behavior of the conditional moment estimator of dikkala2020minimax, introduced in Section (ref). We can apply Theorem (ref) to obtain the rate at which $\widehat{\theta}_n$ converges:

Next, we use Theorem (ref) to derive the asymptotic variance of the estimator, showing that the estimator is in inefficient in general. For this purpose, it suffices to only consider the simpler parametric setting.

chamberlain1987asymptotic derived the efficiency bound for the parametric conditional moment setting, corresponding to the smallest (in a p.s.d. sense) $\sqrt{n}$-asymptotic variance for any unbiased estimator. It is given by the covariance matrix: $$V_*=\mathbb{E}\left[\mathbb{E}\left[\nabla_{\theta_*}m(X,\theta_*)'|Z\right]\mathbb{E}\left[m(X,\theta_*)m(X,\theta_*)'|Z\right]^{-1}\mathbb{E}\left[\nabla_{\theta_*}m(X,\theta_*)|Z\right]\right]^{-1}$$ Note that $V\neq V_*$ in general, implying that $\widehat{\theta}_n$ is an inefficient estimator. By extension, this also applies to the Reinforcement Learning algorithm of Example (ref). Comparing the GMM objective of Example (ref) - which is known to be efficient in the unconditional moment setting - to the population objective of the present example, this may be unsurprising: in contrast to GMM, the population objective of dikkala2020minimax corresponds to a regular $\ell^2$ norm, without the inverse covariance weighting which is crucial for asymptotic efficiency in the unconditional case. Generalizing GEL to the conditional moment setting by replacing the constant adversary with a neural network $\Lambda_n$, metzger2021 therefore proposes the A-estimator given by: $$\inf_{\theta\in\Theta_n}\sup_{\lambda\in\Lambda_n}\mathbb{E}_n\left[-f_*\left(m(X,\theta)'\lambda(Z)\right)\right]$$ which nests a simplified variant of variationalMM2020 for $D_f=\chi^2$, and for $D_f=D_{KL}$ can be viewed as alternative to the Kernel approach of kitamura2004empirical. metzger2021 provides a similar information theoretic foundation as the GEL estimator and - building on the theory developed in the present paper - derives the convergence rates and asymptotic efficiency of this estimator, where $\Theta_n$ may contain unknown functions which are modeled as neural networks.

Estimating Riesz Representers

Finally, we show that Theorem (ref) can be used to quickly derive the convergence rates of chernozhukov2020adversarial's adversarial estimator for Riesz representers, which we introduced in Section (ref).

This result clarifies that the Riesz representer of chernozhukov2020adversarial can similarly benefit from the adaptivity properties of neural networks, which yield faster rates for our target classes if $d_*<D$. In combination with their Lemma 17, this implies that compared to other non-parametric sieves, neural networks guarantee the asymptotic normality of the orthogonalized estimator $\Tilde\phi_n$ under weaker conditions on smoothness and $D$. Since the normality of $\Tilde\phi_n-\phi_*$ is of primary interest and already follows from chernozhukov2020adversarial's Lemma 17 given our convergence rates, we refrain from deriving it for arbitrary functionals $\widehat\phi_n(g)=\mathbb{E}[\widehat{\theta}_n(x)g(x)]$, although it would be possible to use Theorem (ref) to derive $\sqrt{n}(\widehat\phi_n(g)-\phi(g))\overset{d}{\to}\mathcal{N}(0,V_g)$ for some $V_g$ for example.

Conclusion

We characterize the general class of adversarial estimators {\it (`A-estimators')}, subsuming many estimators independently proposed in the fields of econometrics and machine learning. Our unified framework suggests interesting commonalities between A-estimators: their adversary is always Neyman-orthogonal with respect to the main model, guaranteeing that its estimation errors have no first-order asymptotic impact on the estimated model. Most objectives have versions which asymptotically minimize an $f$-divergence criterion and are asymptotically efficient. Typically, A-estimators adaptively learn how to optimally emphasize the restrictions implied researcher's estimation assumptions, performing particularly well when this set is large. This makes them a promising framework for incorporating machine learning methods into causal inference, where even simple target parameters often satisfy a continuum of restrictions. We characterize the convergence rates of A-estimators, as well as the asymptotic normality of smooth functionals of their parameters. We also provide low-level analogues of these results for semi-parametric models, in which unknown functions are approximated by deep neural networks. Our convergence and normality results also extend the theory of neural network M-estimators, as a special case: building on recent results in approximation theory, our neural network converge rates exhibit a reduced curse of dimensionality for more general losses than previously examined, which hold uniformly over a second parameter space. Our normality result overcomes a problem previously posed by the non-convexity of neural network sieves, showing that a particular regularization, combined with under-smoothing, can be used to satisfy a strong, high-level approximation error condition which the literature left hitherto unverified.