Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
65,265 characters · 16 sections · 43 citation commands
Adapting to Misspecification
Empirical research is typically characterized by a robustness-efficiency tradeoff. The researcher can either invoke strong assumptions to motivate an estimator that is precise, but sensitive to violations of model assumptions, or they can employ a less precise estimator that is robust to these violations. Familiar examples include the choice of whether to add a set of controls to a regression, whether to exploit over-identifying restrictions in estimation, and whether to allow for endogeneity or measurement error in an explanatory variable.
Decisions of this nature are often approached with a degree of pragmatism: imposing a false restriction may be worthwhile if doing so yields improvements in precision that are not outweighed by corresponding increases in bias. While precision is readily assessed with asymptotic standard errors, the measurement of bias is less standardized. A popular informal approach is to conduct a series of “robustness exercises,” whereby estimates from models that add or subtract assumptions from some baseline are reported and examined for differences. While potentially informative about the presence of bias, it is often unclear how the results of such exercises should be used to refine baseline estimates of the parameter of interest.
One answer, found often in econometrics textbooks, is to use a specification test to select a model. Doing so yields a pre-test estimator that equals the estimator of the restricted model when the specification test fails to reject, and is otherwise equal to the estimator of the unrestricted model. The pre-test estimator offers a form of asymptotic insurance against bias: as the degree of misspecification grows large relative to the noise in the data, the test rejects with near certainty. Yet when biases are modest, as one might expect of models that serve as useful approximations to the world, the cost of this insurance in terms of increased variance can be exceedingly high.
In this paper we explore an alternative to specification testing: adapting to misspecification.\footnote{An interactive Shiny application implementing our proposed estimator is available online at \url{https://lsun20.github.io/MissAdapt/}.} Adaptive estimation provides a systematic approach to exploiting the assumptions of the restricted model as efficiently as possible while acknowledging the possibility that the restriction in question is misspecified. Consider an oracle who knows a bound on the extent to which the restricted model is misspecified, allowing them to combine the estimates from the restricted and unrestricted models in a way that minimizes maximum risk. An adaptive estimator is one that comes as close as possible to achieving this oracle benchmark without using prior knowledge of the magnitude of misspecification.
We show that adaptive estimators can be computed by solving a weighted minimax problem. While the resulting optimally adaptive estimator does not have a closed form, an analytic soft-thresholding estimator can be tuned to yield comparable performance. This adaptive soft-thresholding estimator can be interpreted as a smoothed version of the pre-test estimator utilizing a critical value that depends on the correlation between the restricted and unrestricted estimators. The near-optimality of adaptive soft-thresholding contrasts with the performance of pre-test estimators, which perform poorly under moderate amounts of misspecification.
Both the optimally adaptive and adaptive soft-thresholding estimators are easily computed using information that is routinely reported in robustness checks. In the case where the restricted estimator is efficient under the restricted model, the estimators can be computed from published point estimates and standard errors alone. The adaptive soft-thresholding estimator can also be obtained via a particular sort of lasso regression tibshirani1996regression that may be of independent interest in other low-dimensional settings.
To illustrate the advantages of adapting to---rather than testing for---misspecification, we revisit two empirical examples where questions of model specification arise. Our leading example, which we return to throughout the paper, is drawn from de_chaisemartin_two-way_2020's reanalysis of gentzkow_effect_2011, in which a two-way fixed effects estimator that exhibits negative weights in many periods is compared to a more variable convex weighted estimator. A second example, taken from angrist_does_1991, compares an ordinary least squares (OLS) estimate of the returns to schooling to an instrumental variables (IV) estimate. We argue that extra care is required in this example because the IV estimate is orders of magnitude less precise than OLS. Online (ref) provides an additional example, drawn from lalonde1986evaluating, illustrating the problem of estimating the effects of job training using a mix of control groups whose credibility can be ranked ex-ante. In all of the above examples, adapting between models is found to yield a more attractive balance between efficiency and robustness than selecting a single model via pre-testing, with the adaptive soft-thresholding estimator performing especially well.
Our analysis builds on early contributions by hodges_use_1952 and bickel_minimax_1983,bickel_parametric_1984 who consider families of robustness-efficiency tradeoffs defined over pairs of nested models. We extend this work by considering a continuum of models, indexed by different degrees of misspecification. A large statistics literature considers the problem of adaptation, defined as the search for an estimator that performs nearly as well as an oracle with additional knowledge of the data generating process. We focus on the case where proximity to oracle performance is measured in terms of the ratio of actual to oracle risk, which mirrors the definition used in tsybakov_pointwise_1998 and leads to simple risk guarantees and statements about relative efficiency. To introduce the core ideas, we begin with a simple introductory example.
In this section, we illustrate our proposal at a high level via an empirical example, postponing the details to later discussion. gentzkow_effect_2011 studied the effects of newspapers on voter turnout in US presidential elections using a two-way fixed effects (TWFE) model estimated in first differences by least squares. de_chaisemartin_two-way_2020 showed that in settings featuring staggered adoption, like the one studied by gentzkow_effect_2011, TWFE estimators identify potentially non-convex combinations of average treatment effects over time and across adoption cohorts.
Suppose the target parameter $\theta$ is the average effect of changing newspaper access on voter turnout in counties exhibiting a change in the number of newspapers. Let $Y_R$ denote the TWFE estimator used by gentzkow_effect_2011 and $Y_U$ the estimator of $\theta$ proposed by de_chaisemartin_two-way_2020. In the presence of treatment effect heterogeneity, $Y_R$ likely identifies a different parameter, implying an unknown bias $b=E[Y_R]-\theta.$ In contrast, $Y_U$ is unbiased for $\theta$. However, when treatment effect heterogeneity is mild, $Y_R$ may exhibit negligible bias and substantially lower variance than $Y_U$, yielding a non-trivial robustness-efficiency tradeoff.
The value of $Y_R$ reported by gentzkow_effect_2011 implies that an additional newspaper raises voter turnout by $0.26$ percentage points, with a standard error of $\sigma_R=0.09$. The unrestricted estimator $Y_U$ evaluates to 0.43, with a standard error of $\sigma_U=0.14$. Suppose that $Y_U$ and $Y_R$ are normally distributed with standard deviations given by these standard errors, an approximation that can be formally justified using a local asymptotic misspecification framework. The difference $Y_O=Y_R-Y_U$ gives a noisy estimate of the bias $b$. To further simplify the example, suppose that $\operatorname{cov}(Y_R,Y_O)=0$. This condition, which seems to be very nearly satisfied in the data, implies that $Y_R$ is efficient under the constraint $b=0$. Consequently, the variance of $Y_O$ is $\sigma_O^2=\sigma_U^2-\sigma_R^2.$ The test statistic that forms the basis for standard “over-identification” tests of specification is $T_O=Y_O/\sigma_O.$
To compare these estimators, consider their mean squared error (MSE), which will be our preferred measure of risk. Since $Y_U$ is unbiased, its MSE is equal to its variance $\sigma_U^2=(0.14)^2$. In contrast, the MSE of the restricted estimator depends on its bias $b$: $E[(Y_R-\theta)^2]=b^2+\sigma_R^2=b^2+(0.09)^2$. Figure (ref) plots the MSE of the unrestricted and restricted estimators as functions of the unknown bias $b$. To ease visual interpretation both risk functions have been divided by $\operatorname{var}(Y_U)$, which normalizes the risk of $Y_U$ to $1$.
When $b=0$, using $Y_R$ instead of $Y_U$ yields a decrease in MSE from $(0.14)^2$ to $(0.09)^2$. The price paid for this improvement in MSE at $b=0$ is that the MSE can be much larger than $(0.14)^2$ when $b\ne 0$. Tradeoffs of this nature are unavoidable because $Y_U$ is admissible: no other estimator has lower MSE for all $b$. The goal of adaptive estimation is to resolve this tradeoff by balancing efficiency when $b$ is close to zero against robustness when $b$ is large.
Given a bound $B\ge 0$ on the bias magnitude $|b|$, one can compute the estimator that is minimax over the restricted parameter space $(\theta,b)\in \mathbb{R}\times [-B,B]$, a procedure we refer to as the $B$-minimax estimator. The $B$-minimax estimator $\hat\theta(Y_U,T_O;B)=Y_R - \sigma_O \delta(T_O;B)$ adjusts the restricted estimator by an estimate $\delta(T_O;B)$ of its bias constructed by smoothly shrinking $T_O$ towards zero, yielding output in the interval $[-B/\sigma_O,B/\sigma_O].$ Figure (ref) plots the risk function of the $B$-minimax estimator for $B\in\{\sigma_O,4\sigma_O\}$. As a benchmark, we also plot the risk function of an oracle estimator computed using prior knowledge of the best possible bound $B=|b|$.
Note that if the posited bound $B$ is set lower than the true bias magnitude $|b|$, $B$-minimax estimation can yield very large MSE. An alternative to guessing a bound $B$ is to use the data to infer a likely value of $|b|$. Then one can estimate $\theta$ optimally subject to the estimated bias magnitude. The pre-test estimator described in the introduction uses $Y_U$ when $|Y_O|>1.96 \sigma_O$ and otherwise relies on $Y_R$. Unfortunately, the risk function of the pre-test estimator, plotted in Figure (ref), is quite large for moderate values of $b$, reflecting the cost of using the data “twice” in a non-smooth fashion.
Adaptive estimators, by contrast, use the data to directly mimic the oracle's risk function. The optimally adaptive estimator is the estimator that comes closest to matching the oracle's risk function, where distance is measured in terms of the maximum ratio of actual to oracle risk across all bias levels, a metric that we term the adaptation regret. Like the $B$-minimax estimator, the optimally adaptive estimator ${\hat\theta^{\ast}}(Y_U,T_O)=Y_R - \sigma_O \delta^*(T_O;1-\sigma_R^2/\sigma_U^2)$ uses $T_O$ to adjust the restricted estimator for bias; however, it depends on $\sigma_R^2/\sigma_U^2$, which captures the efficiency of $Y_U$ relative to $Y_R$, rather than on an ex-ante bound $B$. Though the function $\delta^*(\cdot;1-\sigma_R^2/\sigma_U^2)$ lacks an analytic closed form, a simple soft-thresholding estimator can be tuned to approximate it closely.
Like the pre-test estimator, the resulting adaptive soft-thresholding estimator is equal to $Y_R$ if $|Y_O/\sigma_O|$ is less than some threshold value $\lambda$. However, rather than switching discontinuously to $Y_U$ when $|Y_O|>\lambda \sigma_O$, the soft-thresholding estimator “shrinks” the unrestricted estimator towards the restricted estimator by $\lambda$ standard errors of the bias estimate. The optimal threshold is a decreasing function of the ratio $\sigma_R^2/\sigma_U^2$. In this example, $\sigma_R^2/\sigma_U^2= 0.41$, implying $Y_U$ is only 41% as efficient as $Y_R$ when $b=0$. The corresponding optimal threshold is $\lambda=0.64$, far below the traditional 1.96 value used for pre-testing.
The risk function of the optimally adaptive estimator and its soft-thresholding approximation are shown in Figure (ref). The MSE of the optimally adaptive estimator is never more than 44% above the oracle MSE, which is the best that can be achieved. The adaptive soft-thresholding estimator has an MSE that is never more than 46% above the oracle. When $b=0$, these adaptive estimators achieve substantially lower MSE than $Y_U$. Conversely, when $|b|$ is large, they exhibit modestly higher MSE than $Y_U$. The pre-test estimator also achieves near oracle MSE levels when $b=0$. However, when $|b|\approx1.96\sigma_O$, its MSE is 118% percent above the oracle MSE and 75% above the MSE of $Y_U$.
Both the adaptive estimator and its $B$-minimax counterparts can be thought of as Bayes estimators motivated by particular least favorable priors. Figure (ref) depicts the least favorable priors utilized by the $B$-minimax estimator for two values of $B$ along with the least favorable prior of the adaptive estimator. All three priors are discrete, symmetric about zero, and decreasing in $|b|$. The $B$-minimax priors have support on $[-B/\sigma_O,B/\sigma_O]$ but involve more than two mass points when $B$ is large, which yields statistical uncertainty about both the magnitude and sign of the bias.
The estimators motivated by these three priors will all tend to yield lower MSE than $Y_U$ when the true bias magnitude $|b|$ is small. The adaptive prior has the important advantage over $B$-minimax priors of not requiring specification of the bound $B$. Moreover, the adaptive prior is robust: the risk of the optimally adaptive estimator remains bounded as $|b|$ grows large, whereas the risk of a $B$-minimax estimator grows without limit once $|b|$ exceeds the posited bound $B$.
Consider a researcher who observes data or initial estimate $Y$ taking values in a set $\mathcal{Y}$, following a distribution $P_{\theta,b}$ that depends on unknown parameters $(\theta,b)$. Let $E_{\theta,b}$ denote expectation under the distribution $P_{\theta,b}$. We will study possibly misspecified models in a normal or asymptotically normal setting. Results covering more general models are available in a prior version of this paper armstrong2023adapting.
The random variable $Y=(Y_U,Y_R)$ consists of an unrestricted estimator $Y_U$ of a scalar parameter $\theta\in\mathbb{R}$ and a restricted estimator $Y_R$ that is predicated upon additional model assumptions. The additional restrictions required to motivate the restricted estimator make it less robust but potentially more efficient. To capture this tradeoff, we assume that $Y_U$ is asymptotically unbiased for $\theta$, while $Y_R$ may exhibit a bias of $b$ stemming from violation of the additional restrictions. We focus on the case where $Y_R$ is a single scalar-valued estimate, but extensions to vector-valued $b$ are provided in Appendix (ref).
It will often be convenient to work with the quantity $Y_O=Y_R-Y_U$, which gives an estimate of the bias $b$ that features in conventional tests of over-identifying restrictions. We work with the large sample approximation
The variance matrix $\Sigma$ is treated as known. In practice, feasible versions of our procedures can be computed using a consistent estimate of the asymptotic variance matrix. The model ((ref)) arises as from a local asymptotic framework where $\theta$ and $b$ are scaled by the square root of the sample size and $Y_U$ and $Y_R$ are asymptotically normal.
Under the restriction $b=0$, the efficient GMM estimator of $\theta$ is $Y_{R,GMM}$ and its variance is $\sigma^2_{R,GMM}$, where
In the case where $\rho \sigma_U \sigma_O=-\sigma_O^2$, the restricted estimator $Y_R$ and the efficient GMM estimator $Y_{R,GMM}$ coincide because $\operatorname{cov}(Y_R,Y_O)=0$. One can easily compute $\sigma_O^2$ in this case from the contrast $\sigma_U^2-\sigma_R^2$ hausman_specification_1978. Likewise, when $Y_U$ and $Y_R$ are estimated on independent samples, computation is facilitated by the simple relation $\sigma_O^2=\sigma_R^2+\sigma_U^2.$ The relative efficiency of $Y_U$ to $Y_{R,GMM}$ is given by $\sigma_{R,GMM}^2/\sigma_U^2=1-\rho^2$.
An estimator $\hat\theta:\mathcal{Y}\to\mathcal{A}$ maps the data $Y$ to an action $a\in\mathcal{A}$. The loss of taking action $a$ under parameters $(\theta,b)$ is given by the function $L(\theta,b,a)$. While it is possible to analyze many types of loss functions in our framework, we will focus on the familiar case of estimation of a scalar parameter $\theta\in\mathbb{R}$ with $\mathcal{A}=\mathbb{R}$ and squared error loss $L(\theta,b,\hat\theta)=(\hat\theta-\theta)^2$.
The risk of an estimator is given by the function
An estimator $\hat\theta$ is minimax over the set $\mathcal{C}$ for the parameter $(\theta,b)$ if it minimizes the maximum risk over $(\theta,b)\in\mathcal{C}$. We are interested in a setting where the researcher entertains multiple parameter spaces $\mathcal{C}_B$, indexed by $B\in\mathcal{B}$, which may restrict the parameters $(\theta,b)$ in different ways. Define the $B$-minimax estimator as the $\hat \theta$ that is minimax over $\mathcal{C}_B$ and its maximum risk $R^*(B)$ as the $B$-minimax risk:
We will focus on the parameter spaces:
indexed by a scalar bound $B$ on the magnitude of the bias of the restricted estimator. Hence, the set $\mathcal{C}_\infty$ corresponds to the unrestricted parameter space, while $\mathcal{C}_0$ corresponds to the restricted parameter space. Consequently, the $\infty$-minimax estimator (the $B$-minimax estimator when $B=\infty$) is $Y_U$, while the $0$-minimax estimator (the $B$-minimax estimator when $B$=0) is $Y_{R,GMM}$. In the special case where the restricted estimator is fully efficient, the $0$-minimax estimator is additionally equal to the restricted estimator $Y_R=Y_U+Y_O$.
Researchers are often unwilling to commit to a restricted parameter space $\mathcal{C}_B$, either because they lack appropriate prior information or because priors differ among their scientific peers. Relative to an oracle that knows $|b|\leq B$ and is able to compute the $B$-minimax estimator, an estimator $\hat\theta$ formed without reference to a particular parameter space $\mathcal{C}_B$ yields a proportional increase in worst-case risk given by
We refer to $A(B,\hat\theta)$ as the adaptation regret of the estimator $\hat\theta$ under the set $\mathcal{C}_B$. In our main results, risk corresponds to mean squared error. Hence, $(A(B,\hat\theta)-1)\times100$ gives the percentage increase in worst-case MSE over $\mathcal{C}_B$ faced by an estimator $\hat\theta$ relative to the $B$-minimax estimator.
Define the worst case adaptation regret as $A_{\max}(\mathcal{B},\hat\theta)=\sup_{B\in\mathcal{B}}A(B,\hat\theta)$. The lowest possible value $A_{\max}(\mathcal{B},\hat\theta)$ can take is
Following tsybakov_pointwise_1998, ${A^*}(\mathcal{B})$ gives the loss of efficiency under adaptation. An estimator $\hat\theta$ is optimally adaptive if $A_{\max}(\mathcal{B},\hat\theta)={A^*}(\mathcal{B})$. We use the symbol ${\hat\theta^{\ast}}$ to represent such an estimator.
Note that different ways of defining adaptation regret---e.g., in terms of the level increase in risk, rather than the proportional increase---would lead to different optimally adaptive estimators. The proposed definition has the important advantage of being scale invariant: a change of the units in which MSE is measured will not alter the percentage increase in risk over an oracle. However, when $R^*(0)$ is near zero, the optimally adaptive estimator will prioritize minimizing MSE under $b=0$, a difficulty addressed in Section (ref).
We study parameter spaces $\mathcal{C}_B=\mathbb{R}\times [-B,B]$, where the set of values of $B$ under consideration is $\mathcal{B}=[0,\infty]$. Adaptive estimators yield worst case risk near $R^*(B)$ for all $B$, thereby avoiding commitment to a particular choice of $B$. Another way to avoid specifying $B$ is to make the conservative choice $B=\infty$, leading to the $\infty$-minimax estimator $Y_U$. Since $Y_U$ is admissible, the optimally adaptive estimator cannot provide a uniform improvement on $Y_U$ for all $b\in \mathbb{R}$. However, the optimally adaptive estimator does a better job of mimicking the $B$-minimax estimator for small $B$, while also limiting the increase in risk over $Y_U$ in the worst case.
Early work by bickel_parametric_1984 considered adapting over the granular set $\mathcal{B}^{gran}=\{0,\infty\}$. Naturally, it is easier to adapt to the elements of the finite set $\mathcal{B}^{gran}$ than to the infinite set $\mathcal{B}$. Consequently, ${A^*}(\mathcal{B}^{gran})\leq{A^*}(\mathcal{B})$. However, consideration of $\mathcal{B}^{gran}$ may leave efficiency gains on the table for $0<b<\infty$ because $R^*(b) \leq R^*(\infty)$.
In (ref) we develop a stylized model that illustrates the ability of adaptive decisions to foster consensus among “committees” characterized by different sets of beliefs. When the loss of efficiency under adaptation ${A^*}(\mathcal{B})$ is not too large, the committees will agree to jointly follow the optimally adaptive decision because every committee can be compensated for the small increase in maximum risk over their preferred $B$-minimax level. Taking the committees to represent different camps of researchers, the model suggests adaptive estimation can help to forge consensus between researchers with varying beliefs about the suitability of different econometric models.
This section derives the form of the optimally adaptive estimator in our setting. We begin by noting that the problem of computing adaptive estimators can be reduced to that of computing minimax estimators with a scaled loss function.
Plugging in the definition of $R_{\max}(B,\hat\theta)$ along with $\mathcal{B}=[0,\infty]$ and $\mathcal{C}_B=\mathbb{R}\times[-B,B]$, the criterion that the optimally adaptive estimator ${\hat\theta^{\ast}}$ minimizes can be written
where the last equality follows by noting that the double supremum on either side of this equality is over the same set of values of $(B,\theta,b)$. Since $R^*(B)$ is increasing in $B$, the inner supremum is taken at $B=|b|$, which gives the following lemma.
Lemma (ref) shows that finding an optimally adaptive decision can be written as a minimax problem with a weighted version of the original loss function. In particular, ${\hat\theta^{\ast}}$ is found to minimize the maximum (over $\theta,b$) of the objective $\omega(b)R(\theta,b,\hat\theta)=E_{\theta,b}\omega(b)L(\theta,b,\hat\theta(Y))$. Hence, the optimal adaptive estimator corresponds to a minimax estimator under the loss function $\omega(b)L(\theta,b,\hat\theta(Y))$.
According to Lemma (ref), computing adaptive estimators amounts to solving a weighted minimax problem. In our setting, we can further simplify this problem using invariance. We focus here on the case of squared error loss $L(\theta,b,\hat\theta)=(\theta-\hat\theta)^2$. Appendix (ref) provides proofs of the results in this section and covers general loss functions for estimation of the form $L(\theta,b,\hat\theta)=\ell(\theta-\hat\theta)$. It will be useful to transform the data to $(Y_U,T_O)$, where $T_O=Y_O/\sigma_O$ is the $t$-statistic for a specification test of the null that $b=0$. This representation is equivalent to our original setting because $\sigma_O$ is known.
It follows from invariance arguments that both the $B$-minimax estimator and the optimally adaptive estimator take the form
where $Y_{R,GMM}$ is the efficient GMM estimator given in ((ref)) and $\delta:\mathbb{R}\rightarrow\mathbb{R}$ is an estimator of the scaled bias $b/\sigma_O$. Note that, when $b\ne 0$, $Y_{R,GMM}$ exhibits a bias of $-(\rho\sigma_{U}/\sigma_{O}) b$. The estimator in ((ref)) subtracts from the GMM estimate a corresponding estimate $-\rho\sigma_{U} \delta\left(Y_O/\sigma_O\right)$ of this bias term. Estimators in this class were also considered by magnus_estimation_1999 in the context of linear regression.
The following theorem, which is proved in Appendix (ref), describes the particular functions $\delta(\cdot)$ in the class of estimators defined by ((ref)), used by the $B$-minimax and optimally adaptive estimators.
Part ((ref)) of Theorem (ref) establishes that the $B$-minimax estimator relies on an estimator $\delta(T_O;B)$ of the scaled bias $b/\sigma_O$ that is minimax under the bound $|b|\le B$. This minimax estimation problem is called the bounded normal mean problem and has been studied extensively in the literature. We detail the computation of this estimator in Online Appendix (ref). For finite $B/\sigma_O$, the minimax estimator is the posterior mean against a least favorable prior. Figure (ref) illustrates several such priors. When the interval is small, the least favorable prior concentrates at the two endpoints. For larger intervals, it concentrates at a finite number of points within $[-B/\sigma_O, B/\sigma_O]$ casella_estimating_1981. For $B/\sigma_O = \infty$, the minimax estimator is $T_O$.
Theorem (ref)((ref)) states that the optimally adaptive estimator takes the form in ((ref)) with $\delta(\cdot)$ given by $\delta^*(\cdot;\rho^2)$: the solution to a weighted minimax problem over the scaled bias $\tilde b=b/\sigma_O$. Following part (iii) of Theorem (ref), the problem is solved numerically using a discrete approximation to the least favorable prior over $\tilde b$ as in chamberlain_econometric_2000. The least favorable prior distributions reported in Figure (ref) were computed using this approach. The invariance arguments used to derive ((ref)) imply an independent flat prior for $\theta$. To streamline computation, $\delta^*(\cdot; \rho^2)$ is evaluated on a grid of $\rho^2$ values, creating a lookup table. See Online (ref) for details.
One can write the optimally adaptive estimator as a weighted average:
where $w(T_O)=\delta^*(T_O;\rho^2)/T_O$ is a data-dependent weight. We find numerically that the adaptive estimator “shrinks” $T_O$ towards zero, leading the weight $w(T_O) $ to fall between zero and one for all values of $\rho^2$. The data dependent nature of the weight $w(T_O)$ is clearly crucial for the robustness properties of the optimally adaptive estimator. As $T_O$ grows large, less weight is placed on the optimal GMM estimator and more weight is placed on the unrestricted estimator $Y_U$. If one were to commit ex-ante to a fixed (i.e., non-stochastic) weight on $Y_U$ below one, the worst-case risk of the procedure would become unbounded because the optimal GMM estimator can exhibit arbitrarily large bias.
The distribution of an estimator of the form ((ref)) can be derived by noting that $Y_{R,GMM}$ and $T_O$ are independent, with $Y_{R,GMM}\sim N(\theta-b \rho \sigma_U/\sigma_O, \sigma^2_U(1-\rho^2))$ and $T_O\sim N(b/\sigma_O, 1)$. Let $Z_1$ and $Z_2$ denote independent $N(0,1)$ random variables. Substituting $T_O=Z_1+b/\sigma_O$ and $Y_{R,GMM}=\sigma_U\sqrt{1-\rho^2}Z_2+\theta-b\rho\sigma_U/\sigma_O$ into ((ref)) and rearranging terms yields
This representation holds under the distribution for $(Y_U,T_O)$ maintained in ((ref)), which provides an asymptotic approximation under local misspecification. In this asymptotic regime, consistent estimators of $\rho$, $\sigma_U$ and $\sigma_O$ are available via the usual asymptotic variance formulas used in overidentification tests for GMM. In contrast, $b$ gives the limit of the bias of the restricted estimator divided by $\sqrt{n}$ and cannot be consistently estimated. Consequently, it is not possible to consistently estimate the asymptotic distribution of $\hat \theta(Y_U,T_O)$.
For example, the MSE of the estimator $\hat\theta(Y_U,T_O)$ is
Figures (ref) and (ref) of Section (ref) plot this quantity as a function of $\tilde b$ with consistent estimates of $\rho$, $\sigma_U$, and $\sigma_O$ plugged in. However, $\tilde b$ itself cannot be consistently estimated. See leeb_model_2005 for a discussion of these issues in the context of pre-test estimators.
Using ((ref)), one can obtain a $100\cdot (1-\alpha)\%$ CI that is valid under the parameter space $\mathcal{C}_B=\mathbb{R}\times [-B,B]$ for $(\theta,b)$ by using a critical value $c_{\alpha}(\tilde B)=c_{\alpha}(\tilde B;\rho,\delta)$ solving
This critical value can then be used to form the fixed length confidence interval (FLCI) $\left\{\hat\theta(Y_U,T_O) \pm \sigma_U c_{\alpha}(B/\sigma_O)\right\}$ centered at the estimator $\hat\theta(Y_U,T_O)$. To emphasize the dependence on the parameter space $\mathcal{C}_B$ under which coverage is guaranteed, we will refer to such intervals as $B$-FLCIs. For example, one can form the $B$-FLCI centered at the $B$-minimax estimator by using the critical value $c_{\alpha}(B/\sigma_U)$ for this estimator. Setting $B=\infty$, the $\infty$-FLCI centered at the $\infty$-minimax estimator is the usual CI centered at the unrestricted estimator: $\{Y_U\pm z_{1-\alpha/2}\sigma_U\}$. This CI turns out to be larger than the $B$-FLCI centered at the $B$-minimax estimator for finite $B$, reflecting its validity over the larger parameter space $b\in \mathbb{R}$.
One can compute a $B$-FLCI centered at the adaptive estimator by computing the critical value $c_{\alpha}(B/\sigma_O;\rho,\delta^*(\cdot;\rho^2))$ for the adaptive estimator. Unfortunately, it can be shown formally that any CI that is valid for all $b\in \mathbb{R}$ must have average length close to the length $2z_{1-\alpha/2}\sigma_U$ of the CI centered at $Y_U$, even if $b$ happens to be close to zero armstrong_sensitivity_2021. In light of this impossibility result, it is reasonable to report alongside an adaptive estimate the critical values for a $0$-FLCI and $\infty$-FLCI, thereby summarizing the range of critical values needed to guarantee coverage under different assumptions. When $|\rho|$ is large, the critical value for a $0$-FLCI will be far below the usual 1.96 benchmark for a 95% test. Conversely, the corresponding critical value for a $\infty$-FLCI interval will be much larger than 1.96, reflecting the inherent tradeoffs involved in centering the CI around the adaptive estimator rather than the unbiased estimator. cai_adaptive_2005 discuss analogous tradeoffs involving centering in the context of nonparametric estimation.
An alternate approach, which we explore in our main empirical example, is to construct a $B$-FLCI for some intermediate value of $B$ and report both its worst and best case coverage. Researchers who are open to trading off some worst-case coverage for a shorter CI or enhanced best-case coverage might find an interval centered around an adaptive estimator, offering coverage (say) between 90% and 97%, more appealing than a longer interval centered around $Y_U$ that consistently provides 95% coverage. This interval could also be preferable to a slightly shorter 90% CI centered around $Y_U$, as the additional 7 percentage points of potential coverage may be more valuable than a modest reduction in length.
While the optimally adaptive estimator is trivial to implement once the solution is tabulated, it lacks a simple closed form. To reduce the opacity of the procedure, one can replace the term $\delta(T_O)$ in ((ref)) with an analytic approximation. A natural choice of approximations for $\delta(T_O)$ is the class of soft-thresholding estimators, which are indexed by a threshold $\lambda\ge 0$ and given by
We also consider the class of hard-thresholding estimators, which are given by
Note that hard-thresholding leads to a simple pre-test rule: use the unrestricted estimator if $|T_O|>\lambda$ (i.e. if we reject the null that $b=0$ using critical value $\lambda$) and otherwise use the GMM estimator that is efficient under the restriction $b=0$. The soft-thresholding estimator uses a similar idea, but avoids the discontinuity at $T_O=\lambda$.
A third estimator, which we will call the empirical risk minimizer (ERM), takes the form $\delta_{ERM}(T_{O})=\frac{T_{O}^{2}}{T_{O}^{2}+1}\cdot T_{O}$. The ERM estimator, which was proposed by de_chaisemartin_empirical_2020, minimizes the estimated risk of the weighted average between $Y_{U}$ and $Y_{R,GMM}$. The ERM can be generalized to a broader class of estimators $\delta_{ERM,\lambda}(T_{O})=\frac{T_{O}^{2}}{T_{O}^{2}+\lambda}\cdot T_{O}$, which was briefly considered in Magnus_Estimation_2002. We can optimize $\lambda$ for the worst-case adaptation regret given a specific value of $\rho^{2}$, which yields the adaptive ERM estimator.
To compute the adaptive ERM estimator along with the hard and soft-thresholding estimators that are optimally adaptive in these classes of estimators, we numerically minimize ((ref)) over $\lambda$ as explained in Online Appendix (ref). We plot the respective optimal thresholds in Figure (ref), which are only a function of the relative efficiency $\sigma_{R,GMM}^2/\sigma_U^2=1-\rho^2$. We will be especially interested in the optimal soft-threshold, which can be closely approximated using the formula $\lambda= 0.45 - 0.24 \cdot \ln(1-\rho^2)$ for $\rho^2\in(0.002,0.99)$.
Figure (ref) plots the optimally adaptive and soft-thresholding estimators of the scaled bias as functions of $T_O$. To ease visual inspection of the differences between these estimators, they have been plotted over the restricted range [-3,3]. These functions depend on the data only through the estimated value of $1-\rho^2$, which takes the value 0.41 here, as in the two-way fixed effects example introduced in Section (ref). The optimal soft-threshold $\lambda$ yielding the lowest worst cast adaptation regret in this example is 0.64. The optimally adaptive, adaptive ERM, and soft-thresholding estimators continuously shrink small values of $T_O$ towards zero. However, the soft-thresholding estimator sets all values of $|T_O|$ less than $0.64$ to zero, while the optimally adaptive and adaptive ERM estimators avoid flat regions. In contrast to the continuous nature of these adaptive estimators, a conventional pre-test using $\lambda=1.96$ exhibits large discontinuities at the hard-threshold. The pre-test choice of $\lambda=1.96$ differs from the value that minimizes worst-case adaptation regret, which in this example is 1.43.
Like the optimally adaptive estimator ${\hat\theta^{\ast}}$, the worst-case adaptation regret of the adaptive soft and hard-thresholding estimators depends only on $1-\rho^2$. We report comparisons between these estimators in our empirical applications in Section (ref). As discussed in Online Appendix (ref), soft-thresholding yields nearly optimal performance for the adaptation problem relative to ${\hat\theta^{\ast}}$ in a wide range of settings. In contrast, hard-thresholding typically exhibits both substantially elevated worst case adaptation regret and worst case risk driven by the possibility that the scaled bias has magnitude near $\lambda$. The adaptive ERM estimator generally exhibits slightly higher worst case risk and adaptation regret than the soft-thresholding estimator but exhibits lower risk when the bias is very large.
Our finding that soft-thresholding is nearly optimal for adaptation mirrors the findings of bickel_parametric_1984 for the case where the set $\mathcal{B}$ of bounds $B$ on the bias consists of the two elements $0$ and $\infty$. Magnus_Estimation_2002 reports that soft-thresholding optimizes a related regret problem over a certain class of estimators indexed by two scalar parameters. While soft-thresholding is perhaps the simplest way of achieving near-optimal performance for adaptation, other generalizations of thresholding estimators johnstone_gaussian_2019 have been found to have similar risk properties to soft-thresholding, and may also perform well in our setting.
As detailed in Appendix (ref), the soft-thresholding estimator is numerically equivalent to a generalized lasso estimator tibshirani1996regression applied to a dataset comprised of the restricted and unrestricted estimates. The regressors are a constant and an indicator for the restricted estimate, the coefficient on which measures the bias $b$. The lasso penalty shrinks the bias estimate towards zero and depends only on the soft-threshold $\lambda$. Hence, the adaptive soft-threshold provides an optimal tuning of lasso for low-dimensional settings in which interest centers on a scalar parameter. This exact tuning contrasts with high-dimensional settings where existing tuning methods typically only offer rate results.
If the loss of efficiency under adaptation $A^*(\mathcal{B})$ is large, both the optimally adaptive estimator and its soft-thresholding approximation will possess worst case risk far above the oracle minimax risk, which limits their practical appeal. As we show in Online Appendix (ref), $A^*(\mathcal{B})$ will tend to be large when $|\rho|$ is large, which corresponds to settings where $Y_R$ is orders of magnitude more precise than $Y_U$.
In such cases, it may be attractive to temper the degree of adaptation that takes place by restricting attention to estimators that exhibit worst case risk no greater than a constant $\bar R$. Online Appendix Section (ref) details how to compute such a constrained adaptive estimator. As noted by bickel_parametric_1984 in his analysis of the granular case where $\mathcal{B}=\{0,\infty\}$, it is often possible to greatly improve the risk at $b=0$ relative to the unbiased estimator $Y_U$ in exchange for modest increases in risk when $b=\infty$. Similarly, we find that setting $\overline R$ to $50\%$ above the risk of $Y_U$ yields large efficiency improvements when $b$ is small.
The constrained adaptive estimator bears some similarity to the ERM estimator. de_chaisemartin_empirical_2020 prove that the maximal risk decrease of $\delta_{ERM}$ relative to the risk of the unbiased estimator is larger than the maximal risk increase of $\delta_{ERM}$ relative to the unbiased estimator. Through numerical calculations reported in a prior version of this paper armstrong2023adapting, we find that this property holds for the constrained soft-thresholding version of our estimator so long as $\overline R$ is less than $70\%$ above the risk of $Y_U$. Remarkably, the property holds even for unconstrained soft-thresholding ($\overline R=\infty$) so long as $\rho^2$ is less than 0.86.
We now consider two empirical examples where questions of specification arise and examine how adapting to misspecification compares to pre-testing and other strategies such as committing ex-ante to either the unrestricted or restricted estimator. A third example, provided in Online (ref), considers a multivariate adaptation problem with two restricted models and corresponding bias estimates.
Returning to the example introduced in Section (ref), Table (ref) reports the realizations of $(Y_U,Y_R)$ and their standard errors, which exactly replicate those given in Table 3 of de_chaisemartin_two-way_2020 after dividing by 100. The estimated variance of $Y_O$ is closely approximated by the difference in squared standard errors between $Y_U$ and $Y_R$, suggesting $Y_R$ is nearly efficient. Hence, the downstream GMM, adaptive, and soft-thresholding estimators could have been accurately approximated using only the published point estimates and standard errors. In contrast to the analysis in Section (ref), we treat $Y_{R,GMM}$ rather than $Y_R$ as the efficient estimator, resulting in small differences from the previously reported downstream results. Standard errors are not reported for the soft-thresholding, adaptive, or pre-test estimators because the variability of these procedures depends on the unknown bias level $b$.
Both $Y_R$ and $Y_{R,GMM}$ exhibit standard errors roughly 35% below that of $Y_U$. Consequently, relying solely on the convex-weighted estimator $Y_U$ exposes the researcher to a large worst-case adaptation regret of 145%. Though the realized value of $Y_U$ is nearly twice as large as that of $Y_{R,GMM}$, the two estimators are not statistically distinguishable from one another at the 5% level. Hence, a conventional pre-test suggests ignoring the perils of negative weights and confining attention to $Y_{R,GMM}$ on account of its substantially increased precision. The worst case MSE of the pre-test estimator, which exhibits a hump shaped risk profile similar to that depicted in Figure (ref), is 87% higher than the MSE $\sigma_U^2$ of $Y_U$. Pre-testing also yields sizable worst-case adaptation regret reflecting the possibility that the test selects the inefficient $Y_U$ when $b=0$.
In contrast to the pre-test estimator, both the optimally adaptive estimator and its soft-thresholding approximation place substantial weight $w(T_O)$ on the convex estimator, yielding estimates roughly 60% of the way towards $Y_U$ from $Y_{R,GMM}$. This phenomenon owes to the fact that with $T_O=-1.75$ both estimators detect the presence of a non-trivial amount of bias in $Y_R$. We can easily compute the soft-thresholding bias estimate from the figures reported in the table as $(-1.75+.64)\times-0.77\times0.14\approx 0.12$, suggesting that $Y_{R,GMM}$ exhibits a bias of roughly 50%. Balancing this bias against the estimator's increased precision leads the soft-thresholding estimator to essentially split the difference between the convex and non-convex weighted estimators.
By construction, the adaptive estimator exhibits lower worst case adaptation regret than the soft-thresholding estimator but the differences are quantitatively trivial. However, the soft-thresholding estimator exhibits meaningfully lower worst case risk than the adaptive estimator. Though the two estimators happen to yield identical estimates ex-post in this example, the ex-ante risk properties of the adaptive soft-thresholding estimator arguably commend it over the optimally adaptive estimator.
The ERM estimator of de_chaisemartin_empirical_2020 yields lower worst case risk than soft-thresholding but substantially larger adaptation regret. Optimizing the ERM threshold to minimize adaptation regret yields worst case risk equivalent to the soft-thresholding estimator but higher adaptation regret. Of the estimators considered, soft-thresholding offers the most attractive tradeoff between worst case risk and adaptation regret.
Confidence Intervals Table (ref) reports the best case and worst case coverage of a series of confidence intervals. The first two columns of Panel A show that the usual 95% confidence interval centered around the unbiased estimator has proper size, while a naive CI centered around the restricted estimator has best case coverage of 95% and worst case coverage of 0% attributable to the potentially unlimited bias of the restricted estimator. Relying on a pre-test to select one of these two confidence intervals yields a minimum coverage level of 67%. By contrast, centering a CI around the optimally adaptive estimator using the standard error of the unbiased estimator yields best case coverage of 98% and worst case coverage of 90%. Centering around the soft-thresholding estimator yields even more favorable results, raising the worst case coverage to 93%.
Panel B of Table (ref) considers $B$-FLCIs centered around the adaptive estimators. A $0$-FLCI centered around the optimally adaptive estimator has a half length of only about $1.54\sigma_U$ (as opposed to the traditional $1.96\sigma_U$ utilized in Panel A) but exhibits worst case coverage of 80%. Centering around the soft-thresholding estimator yields a slightly longer interval, which improves minimum coverage to 87%. The third row of Panel B shows the coverage of a $\sigma_O$-FLCI centered around the optimally adaptive estimator, which yields modestly longer CI but lowers worst case coverage to 86%. Again, centering at the soft-thresholding estimator raises worst case coverage slightly, in this case to 90%. Finally, we approximate an $\infty$-FLCI by setting $B=9\sigma_O$, which yields very conservative intervals with half-lengths exceeding $2.1\sigma_U$.
The simplicity and robustness of intervals based upon an adaptive estimator $\pm 1.96\sigma_U$ make them an attractive option. For researchers who seek shorter intervals, the $\sigma_O$-FLCI centered around the soft-thresholding estimator seems to offer a reasonable mix of worst and best case coverage. Notably, all of these options offer substantially higher worst case coverage than pre-testing, which remains widespread in applied research.
Our second example, which is meant to highlight the limits of optimal adaptation, comes from angrist_does_1991's seminal analysis of the returns to schooling using quarter of birth as an instrument for schooling attainment. Table (ref) replicates exactly the estimates reported in angrist_does_1991 for men born 1930-39. $Y_U$ gives the Wald-IV estimate of the returns to schooling using an indicator for being born in the first quarter of the year as an instrument for years of schooling completed, while $Y_R$ gives the corresponding OLS estimate. Neither estimator controls for additional covariates.
The first stage relationship between quarter of birth and years of schooling exhibits a z-score of 8.22, suggesting an asymptotic normal approximation to $Y_U$ is likely to be highly accurate. We follow the original study in assuming homoscedasticity, in which case OLS ($Y_R$) is known to be the asymptotically efficient GMM estimator under exogeneity.
While the IV estimator accounts for endogeneity, it is highly imprecise, with a standard error two orders of magnitude greater than OLS. Consequently, the maximal regret associated with using IV instead of OLS is extremely large, as $Y_U$ is only 0.02% as efficient as $Y_R$ when exogeneity holds. IV and OLS cannot be statistically distinguished at conventional significance levels, with $T_O\approx-1.3$. The inability to distinguish IV from OLS estimates of the returns to schooling is characteristic not only of the specifications reported in angrist_does_1991 but of the broader quasi-experimental literature spawned by their landmark study card1999causal.
The confluence of extremely large maximal regret for $Y_U$ with a statistically insignificant difference $Y_O$, leads the adaptive estimator, the soft-thresholding estimator, and the pre-test estimator to all coincide with $Y_R$. Despite the agreement of the three approaches, the extremely large adaptation regret exhibited by the optimally adaptive estimator suggests it is unlikely to garner consensus in this setting. While the adaptive and soft-thresholding estimators avoid committing to either $Y_U$ or $Y_R$ before observing the data, they still expose the researcher to more than a 400% increase in maximal risk over $Y_U$. A skeptic concerned with the potential biases in OLS is therefore unlikely to be willing to rely on such an estimator.
If we instead limit ourselves to a 50% increase in maximal risk, the adaptive and soft-threshold estimators yield returns to schooling estimates of 0.080 and 0.085 respectively. While the former estimate is a bit closer to OLS than IV, the latter is approximately halfway between the two. The maximal regret of both these estimators is extremely high, reflecting the potential efficiency costs of weighting $Y_U$ so heavily. These efficiency concerns are likely outweighed in this case by the potential for extremely large biases.
Empiricists routinely encounter robustness-efficiency tradeoffs. The reporting of estimates from different models has emerged as a best practice at leading journals. The methods introduced here provide a scientific means of summarizing what has been learned from such exercises and arriving at a preferred estimate that trades off considerations of bias against variance.
Computing the adaptive estimators proposed in this paper requires only point estimates, standard errors, and the covariance between estimators, objects that are easily produced by standard statistical packages. As our examples revealed, in many cases the restricted estimator is nearly efficient, implying the relevant covariance can be deduced from the standard errors of the restricted and unrestricted estimators.
In line with earlier results from bickel_parametric_1984, we found that soft-thresholding estimators closely approximate the optimally adaptive estimator in the scalar case, while requiring less effort to compute. An interesting topic for future research is whether similar approximations can be developed for higher dimensional settings where the curse of dimensionality renders direct computation of optimally adaptive estimators infeasible.