Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
121,492 characters · 34 sections · 83 citation commands
Individual Shrinkage for Random Effects
\setstretch{1.1}
\hfuzz=14pt
\setstretch{1.2} \parskip 2pt
{\it “Knowing when to borrow and when not to borrow is one of the key aspects of statistical practice" (mallows1982overview)}
Estimating fixed or random effects (RE) and forecasting individual outcomes are core problems in econometrics, with application across different fields of economics. A significant econometric challenge arises when estimators rely on micropanels, as the short time dimension results in imprecise estimates.\footnote{Estimating individual effects and forecasting with micropanels are the goals of several literatures. One example is the vast literature using “value added models" to capture institutional effects, e.g., the effect of teachers on students' test scores Kane:Staiger:08,CFR:2014:a,CFR:2014:b, Angrist17, the effect of neighborhoods on intergenerational mobility CH18, or the effect of physicians on patients' outcomes FHB14. Hull20 discusses how such estimates play a key policy role in the regulation of healthcare and education in the U.S. Other literatures include kline2022systemic, who estimate firm-specific effects in order to analyze bias in firms' hiring decisions and chamberlain1999predictive, who consider forecasting individual incomes for consumption/savings decisions. Macroeconomic panel forecasting also falls into this category if it uses short estimation windows to account for parameter instability (e.g. Liu:2020 forecast banks' revenues after a regulatory change).} This challenge has motivated the widespread adoption of Bayesian shrinkage methods, such as the classical JS61's estimator (JS) and its extension by kwon21 or modern Empirical Bayes methods (EB), which “borrow strength" from other individuals to enhance accuracy. We argue that existing shrinkage methods, while improving aggregate performance, may lead to inaccurate decisions at the individual level. We propose a complementary shrinkage method designed to overcome the “tyranny of the majority" inherent in existing approaches.
Existing shrinkage methods minimize aggregate loss rather than individual loss. However, aggregate performance is not always the primary objective in many economic applications. For instance, when RE are estimated to guide policy interventions targeting specific individuals (e.g., teacher dismissal or hospital reward allocation, as in the value-added literature discussed by Hull20), forecast financial institution distress Liu:2020, or provide personalized financial advice chamberlain1999predictive, the focus is presumably on maximizing the accuracy of decisions for each individual, rather than optimizing aggregate performance.
Some pitfalls of targeting aggregate loss have been recognized in the Bayesian literature. For example, efron1971limiting and mallows1982overview highlight the “relevance" problem of the JS estimator, arguing that it suffers from the tyranny of the majority, by assuming that all other individuals are equally relevant for borrowing strength. Intuitively, JS shrinks individuals by the same amount regardless of their RE, which can result in large bias for outliers, i.e., individuals with RE that are far away from the common mean.\footnote{efron1971limiting suggests tackling the problem by first identifying and then not shrinking outliers; efron2010large proposes to use covariates to identify relevant individuals.} EB relies on stronger assumptions than JS, notably exchangeability and a distributional assumption for the errors. Exchangeability could be violated if a common RE distribution across individuals does not exist, raising concerns about robustness and external validity: RE estimates become dependent on the sample composition rather than reflecting true individual quality. Modern EB methods (e.g., efron2010large, gu2015unobserved,gu2023invidious, Liu:2020, chen2022empirical and koenker2024empirical) have made substantial progress in relaxing distributional assumptions on the prior; however, the impact of error distribution misspecification for EB remains largely unexplored. While there is some evidence that EB could be less susceptible to the tyranny of the majority under correct specification (see, e.g., some simulation results in Liu:2020), we show that a misspecified error distribution can result in large bias, particularly for outliers.
To address these limitations, we propose a shrinkage approach that explicitly targets individual loss. We introduce a class of shrinkage estimators that, similarly to JS, borrow strength from other individuals by shrinking the time series estimators of RE towards a common mean. Unlike existing approaches, however, our shrinkage with individual weights (IW) leverages solely the individual's own past history, rather than the cross-sectional dimension. Our goal is to be agnostic about parameter heterogeneity beyond the common mean, effectively representing the opposite end of the spectrum from existing literature, which assumes a shared parameter distribution across individuals.
While deriving optimal individual shrinkage rules (i.e., oracle weights) based on the Mean Squared Forecast Error (MSFE) criterion is straightforward in theory, the main challenge we address in this paper is constructing feasible (i.e., estimable) weights that perform well despite no restrictions on parameter heterogeneity and short time-series data. To tackle these issues, we employ the Minimax Regret criterion, which is well-suited when little is known about the parameter space. We first derive oracle weights that are minimax regret optimal relative to either the individual time series estimator or the pooled estimator. Although estimating these weights accurately in short samples remains challenging, we demonstrate that imposing bounds on a specific conditional expectation allows us to obtain feasible weights.
Our method does not rely on the strong assumptions inherent in EB methods. While EB necessitates the specification of a complete model with distributional assumptions on the errors, our method is semiparametric. Similarly to JS, IW can be viewed as a best linear rule that minimizes individual loss (whereas JS minimizes aggregate loss). IW does not assume a common RE distribution and thus accommodates richer unobservable heterogeneity, which we show is a crucial ingredient for obtaining accuracy at the individual level. In contrast, JS assumes homogeneous variances of the unobserved RE and errors. kwon21 can be interpreted as an extension of JS that assumes homogeneous variance for the RE but allows for heterogeneous variance of the errors. Standard applications of EB methods tend to assume a common RE distribution but can in principle allow for heterogeneous variances of the errors (e.g., gu2015unobserved and Section 5 of Liu:2020).
In alignment with our reference literature, we focus on a model where individual outcomes are expressed as the sum of RE and idiosyncratic errors. As previously mentioned, we accommodate unrestricted heterogeneity in parameters, while maintaining the assumption that RE share a common mean. This common mean is the point towards which we shrink the estimators/forecasts, representing how we borrow strength. The point of shrinkage is typically assumed to be zero if outcomes are demeaned, otherwise it is approximated by the pooled mean. Consistent with the literature, we also assume independence between RE and errors; however, we impose no restrictions on the relationship between the parameters that characterize their distributions. Notably, the outcomes in our model can also be interpreted as residuals from the first-step estimation of a linear panel data model or a value-added model with homogeneous coefficients for covariates, which may include lagged outcomes. Our framework thus encompasses a broad range of empirically relevant models.
In contrast to existing methods, an advantage of IW is that it does not depend on a large cross-sectional dimension for accuracy, which enables its application in small-sample settings. A drawback of relying on small samples is that it precludes the use of asymptotic behavior for evaluating the performance of IW. We focus instead on finite-sample optimality and on robustness, that is, on showing that IW performs well over the unknown parameter space. Our approach is inspired by manski2019econometrics, who emphasizes evaluation of decision rules by their performance across the parameter space and advocates the Minimax Regret criterion.\footnote{Minimax Regret properties of shrinkage estimators have been studied by magnus2002estimation and hansen2015shrinkage. However, their setting is different from ours in that they consider the problem of estimating the mean of normally distributed variables with known variance in the univariate case (magnus2002estimation) and in the multivariate case (hansen2015shrinkage). In addition, there is some literature applying the Minimax Regret criterion to panel data in order to address different questions: handling missing data in sample design Dominitz:Manski:2021 and forecasting discrete outcomes under partial identification or other concerns CMS:20. Their focus is also distinct from ours.} manski2019econometrics promotes this criterion as a robust, practical, and cautious decision-making framework, particularly effective under a high level of uncertainty. In our context, the Minimax Regret criterion not only provides a decision-theoretic foundation for the proposed class of shrinkage estimators, but it also aids in selecting the optimal weights in practice.
Since in our model the estimator of RE coincides with the forecast of the individual outcome, we can equivalently focus the discussion on estimation or forecasting. The theoretical results in the paper focus on forecasting, but we present the analogous results for estimation in the Appendix. We show that any IW forecast is Minimax Regret optimal over the parameter space, relative to using either the time series forecast or the common mean. In addition, IW is also optimal in terms of MSFE if we restrict attention to the region of the parameter space where the time series forecast and the common mean are equally accurate. Keeping all else equal, an additional improvement can be obtained under a key assumption that requires the IW weights to be genuine functions of the RE (in addition to not being “pathological", in the sense of shrinking outliers more than RE near the mean, which would exacerbate the tyranny of the majority phenomenon). JS, for example, does not satisfy this key assumption because its weights are based on cross-sectional information and thus are not functions of the RE. Finally, we show that the accuracy improvement of IW under the key assumption is larger the heavier the tails of the RE distribution. In other words, IW is particularly advantageous in the presence of heavy tails because the weights implicitly relate the amount shrinkage to how “far" the RE is from the common mean. This is what allows IW to overcome the tyranny of the majority.\footnote{The robustness of our approach is rooted in the focus on individual rather than aggregate accuracy. It is however worth noting that IW could dominate existing methods also in terms of aggregate accuracy. Whether this occurs in a given application depends on the distribution of the parameters across individuals - which our individual-level analysis leaves unrestricted - and also on the unknown tail properties of the RE distribution. The simulations and empirical applications offer some insights in this respect.}
We present three types of feasible weights for IW, all of which satisfy the key assumption: estimates of the oracle weights that are optimal in terms of the individual MSFE (IW-O); “Minimax Regret optimal weights" (IW-MR) and feasible weights based on the (in-sample or out-of-sample) inverse squared forecast error (IW-MSFE), which are equivalent to the weights considered in the forecast combination time-series literature (e.g., bates1969combination, stock1998comparison), in our case computed over a very short time series. These weights offer additional robustness benefits because they do not rely on correct specification of the model and can thus be applied in more general settings. We compare the finite sample performance of all feasible weights and conclude that IW-MR are the preferred weights, closely followed by (in-sample) inverse MSFE weights. Additional simulations illustrate how IW can overcome the tyranny of the majority phenomenon that affects JS.
We present two empirical illustrations. The first revisits the application in kline2022systemic, focusing on gender discrimination in firm hiring. We show that IW-MR delivers different results in terms of estimation and policy implications, compared to the EB procedure used in the paper. IW-MR also dominates in terms of forecasting performance and robustness/external validity. The second application forecasts individual earning residuals using the Panel Study of Income Dynamics. We find that the forecast with the best aggregate accuracy is IW-MR, which tends to assign high shrinkage weights to individuals with earning residuals near the median of the distribution. This application illustrates the potential usefulness of our approach even in terms of aggregate performance and even in highly heterogeneous environments where only a few individuals benefit from borrowing strength.
The rest of the paper is organized as follows. Section (ref) discusses the limitations of existing shrinkage approaches. Section (ref) summarizes our proposed IW method. Section (ref) shows the optimality of IW for the case of forecasting, both in terms of Minimax Regret and of MSFE, in a simplified setting with independent weights and forecasts. Section (ref) derives feasible weights for IW in a simplified setting with independent weights and forecasts. Section (ref) shows simulation results. Section (ref) discusses two empirical applications. Section (ref) offers concluding remarks. Appendix A contains the proofs, Appendix (ref) contains results regarding estimation and Appendix C formally derives the general feasible weights for IW presented in Section (ref).
In this section, we illustrate how existing shrinkage methods that target aggregate loss - JS and EB - may deliver inaccurate individual-level decisions. We discuss the tyranny of the majority phenomenon as well as the effects of violating the exchangeability assumption or misspecifying the error term distribution.
Suppose that the individual outcomes are the sum of independent RE and errors:
where $A_i \sim (0, \lambda_i^2)$ and $U_{i,t} \sim (0, \sigma_i^2)$. Note the common zero mean and the heterogeneous variances. Denote by $\bar{Y}_i$ the $i$th-time series mean, i.e., the Maximum Likelihood Estimator:
We wish to estimate $A = \left(A_1, A_2, ..., A_N\right)$ or equivalently forecast $Y_{i, T+1}$ for each individual $i$ based on information up to time $T$, using a decision rule $\delta = \left(\delta_1, \delta_2, ..., \delta_N\right)$.\footnote{For notational simplicity we suppress the dependence on the sample.} Here we focus on estimation, but similar considerations apply to the forecasting problem.
An optimal rule can minimize individual or aggregate loss:
If one targets individual loss, restricting attention to linear rules delivers the optimal rule:\footnote{This can be obtained by applying the “best linear rule" in equation (9.4), page 129 of efron1973stein, to $\bar{Y}_{i}|A_i\sim(A_i,\sigma_i^2/T)$ and $A_i\sim(0,\lambda_i^2)$.}
The oracle IW rule thus shrinks the MLE towards the common zero mean, using individual-specific weights.
Targeting aggregate loss means using the posterior mean as the optimal rule. Here we briefly present the main Bayesian estimators that have been considered in the literature: the JS estimator of JS61 and EB estimators such as Liu:2020, efron2016empirical, and gu2015unobserved.
In the context of model ((ref)), JS does not require distributional assumptions but assumes homoskedasticity, $\sigma_i^2=\sigma^2$, $\lambda_i^2= \lambda^2$. JS61 and efron1973stein show that the following rule has lower aggregate risk, $\mathbb{E}_{A}L(\delta, A)$, than the MLE:
In practice, the JS weight is estimated by leveraging the cross-sectional dimension, as $\hat{\lambda}^2/(\hat{\lambda}^2+\hat{\sigma}^2/T)$, where $\hat{\sigma}^2/T=1/N \sum_{i=1}^N \left[1/(T-1)\sum_{t=2}^T (Y_{i,t}-Y_{i,t-1})^2\right]/(2T)$ and $\hat{\lambda}^2=1/N \sum_{i=1}^N(\bar{Y_i} - 1/N \sum_{i=1}^N\bar{Y_i})^2-\hat{\sigma}^2/T$. Notice that JS differs from the oracle IW rule in that it imposes constant weights across $i$, resulting in the same amount of shrinkage for all individuals.
EB methods require additional distributional assumptions on the error $U_{i,t}$ and estimate $A_i$ via the posterior mean. Two paradigms exist for computing the posterior mean: “G-modeling" and “f-modeling". In G-modeling, one first estimates the distribution of $A_i$, $G(A)$, leveraging the cross-sectional dimension, and then obtains the EB decision rule as the posterior mean based on the estimated $G(A)$. One approach to G-modeling is efron2016empirical, where the errors are assumed normal and homoskedastic and $G(A)$ is flexibly parameterized by a spline and estimated via deconvolution. Another G-modeling approach is gu2015unobserved, which uses a nonparametric maximum likelihood estimator based on the Kiefer-Wolfowitz approach for mixture models. gu2015unobserved assume normal errors that can be heteroskedastic (treating the variances as random). The G-modeling approach under normality and homoskedasticity results in the following rule:
with $\varphi _{i}\left( \cdot \right) $ the probability density function of a $\mathcal{N}\left(0,\sigma^{2}/T\right)$.
A different approach is f-modeling, which bypasses estimation of $G(A)$ and directly estimates the posterior mean using the Tweedie correction (available under the assumption that errors belong to the exponential family of distributions, see efron2011tweedie). The correction only depends on $G$ through the marginal density of the data, which can be estimated nonparametrically leveraging the cross-sectional dimension. Liu:2020 adopt this strategy, assuming normal and homoskedastic errors (Section 5 of Liu:2020 extends the method to allow for certain forms of heteroskedasticity in the errors), yielding the following rule:
where $l'(\bar{Y}_i)=\frac{\partial}{\partial \bar{Y}_i} \text{log} f(\bar{Y}_i)$ and $f(\bar{Y}_i)$ is the marginal density of $\bar{Y}_i$.
efron1971limiting, mallows1982overview and efron2010large discuss the notion of “relevance", which highlights a key challenge for Bayesian shrinkage methods: knowing which individuals are relevant for estimating a given $A_i$. efron2010large shows that ignoring relevance leads to bias and suggests using covariate information, when available, to define relevant individuals. To see why JS can lead to incorrect decisions at the individual level, assume a constant $\sigma^2_i=\sigma^2$ for simplicity. The individual-level bias incurred by using JS instead of the optimal rule (the oracle IW) is then: $$ Bias_i = \left(\frac{\lambda_i^2}{\lambda_i^2+\sigma^2/T} - \frac{\lambda^2}{\lambda^2+\sigma^2/T}\right)\bar{Y}_i.$$ As long as $\lambda_i^2 \neq \lambda^2$, the bias is large for large values of $\bar{Y}_i$, e.g., for outliers. This can be seen as an illustration of the tyranny of the majority phonomenon discussed by efron2010large.
There is some evidence that EB is less susceptible to the tyranny of the majority phenomenon under correct specification of the error distribution (see, e.g., simulation results reported in Figure 1 in Liu:2020).
Related to the notion of relevance is the concept of exchangeability, which implies that the joint distribution of the data is invariant to permutations of the indices. Here we discuss the implications of the exchangeability assumption made by EB methods, specifically the assumptions of a common RE distribution across individuals combined with i.i.d. observations, which are sufficient to ensure exchangeability. Consider our fully heterogeneous model ((ref)) and maintain the normal errors assumption made by EB: $U_{i,t} \sim \mathcal{N}(0, \sigma_i^2)$. Then, the marginal likelihood is
where $L_i(\cdot | \lambda_i^2 )$ is the $i$-specific distribution of $A_i$ and $H_i(\cdot, \cdot)$ is the $i$-specific distribution of $(\lambda_i^2, \sigma_i^2)$. This likelihood function is too heterogeneous to be embedded in the EB approach, which thus requires additional assumptions. First, one typically assumes homogeneous $\lambda_i^2 \equiv \lambda^2$ and identical distributions $L_i$ and $H_i$ across $i$. Then, the marginal likelihood becomes
where $G( a, \sigma^2 ) := L (a) H (\sigma^2)$ is the unknown but common distribution. Second, EB methods typically assume that the random components $( A_i, \sigma_i^2)$ are i.i.d. over $i$, so that $G$ can be estimated nonparametrically. An EB decision rule is the posterior mean of the estimated $G$. Differences between the likelihoods in ((ref)) and ((ref)) imply that EB might lead to incorrect decision rules in our setting with heterogeneous $\lambda_i^2$ and no exchangeability assumptions.
EB methods rely on a complete model and assume a specific error distribution. The effects of misspecifying such distribution are largely unexplored in the literature. Here we illustrate the impact of misspecifying this distribution on EB estimators based on the Tweedie correction (e.g., Liu:2020) in a simple example.
For simplicity, assume $T=1$ and drop the $t$ subscript. Suppose that $U_{i}$ are distributed as a standardized Gamma distribution with zero mean, unit variance, and skewness $\gamma$.\footnote{Specifically, we consider $f_0(y)$ to be a standardized gamma variable with shape parameter m: $$f_0(y) \sim \frac{\operatorname{Gamma}_m - m}{ \sqrt{m}}$$ with $\operatorname{Gamma}_m $ having density $y^{m-1} \operatorname{exp}(-y)/m!$ for $y \geq 0$. Thus, $f_0(y)$ has mean 0, variance 1, and skewness $\gamma \equiv 2/\sqrt{m}$.} We can then obtain the misspecification bias of the EB estimator when wrongly assuming a standard normal distribution for the error term as:\footnote{Tweedie's formula under normal errors with zero mean and unit variance gives the posterior mean as: $$E[A_i|Y_i] = Y_i+ l'(Y_i),$$ where $l'(Y_i)=\frac{\partial}{\partial \bar{Y}_i} \text{log} f(Y_i)$ and $f(Y_i)$ is the marginal distribution of $Y_i$. Using efron2011tweedie, which shows Tweedie's formula when the error comes from a distribution in the exponential family, a Gamma distribution with shape parameter $m$ (and thus skewness $\gamma \equiv 2/\sqrt{m}$), zero mean and unit variance implies a posterior mean: $$E[A_i|Y_i] = \frac{Y_i+\gamma/2}{1+\gamma Y_i/2}+ l'(Y_i).$$ The bias in ((ref)) is obtained by subtracting the two expressions for $E[A_i|Y_i] $ under the assumption that $l'(Y_i)$ is the same in both cases as it is estimated from the data.}
We thus see that when $|Y_i|$ is large, e.g., for outliers, the misspecification bias is large in absolute value (the bias is approximately $\gamma/2$ when $Y_i$ is near the zero mean). This can be viewed as another manifestation of the tyranny of the majority phenomenon, here due to misspecification.
Our focus on individual loss rather than aggregate loss allows us to overcome the issues described in Section (ref). In particular, we bypass the problem of relevance by only using the individual's past history to form the shrinkage weights instead of using the cross-sectional dimension, thus overcoming the tyranny of the majority. We do not require exchangeability or parametric assumptions on the error term and thus do not suffer from misspecification bias.
Our method is simple to implement. For each individual $i$, we propose shrinkage rules:
The point of shrinkage $\mu$ is either known (e.g. $\mu=0$ if the observations have been demeaned or if $Y_{i,t}$ are residuals from a first step estimation that includes an intercept) or is approximated with the pooled mean, $\mu= \Sigma^N_{i=1}\Sigma^T_{t=1}Y_{i,t}/NT$. $\widehat{Y}^{TS}_{i,T}$ is an estimator of the RE or a forecast of the outcome $Y_{i,T+1}$ that is only based on the time series dimension, typically the MLE: $\widehat{Y}^{TS}_{i,T}= \bar{Y}_{i,T}=\Sigma^T_{t=1}Y_{i,t}/T$. We derive three classes of feasible individual weights $W_{i,T}$, which we report here for convenience for the case where $\widehat{Y}^{TS}_{i,T}=\bar{Y}_{i,T}$.
Estimated Oracle Weights (IW-O):
where $(\cdot)^+$ denotes the positive part.
Minimax Regret Optimal Weights (IW-MR): These are the weights that perform best in our simulations:\footnote{A similar performance in simulations and in the empirical applications is obtained by the following IW-MR rule, based on an alternative unbiased estimator for $\sigma_i^2$:
}
Inverse MSFE Weights (IW-MSFE): These weights are based on the inverse MSFE (either in-sample or out-of-sample). Since they do not rely on the model assumptions, these weights are applicable in more general settings than the model considered in the paper. The in-sample inverse MSFE weights (which are the second best-performing weights in our simulations) are:
For a given choice of $P<T$, the out-of-sample inverse MSFE weights are:
where $\widehat{Y}^{TS}_{i,t-1}$ is the time series mean using data up to time $t-1$ (using all the data available or an arbitrary number of most recent data). For IW-MSFE-OOS, if $\mu$ is not known, it can be approximated with the pooled mean using data up to time $t-1$.
We note that the IW-MR feasible weights in ((ref)) ignore the possible dependence between $W_{i,T}$ and $\widehat{Y}^{TS}_{i,T}$. While we derive theoretical weights that account for such dependence in Appendix C, any attempt to estimate the additional term this introduces into the theoretical weights - a conditional expectation - would likely worsen the performance of the feasible weights. The feasible weights in ((ref)) thus approximate this additional conditional expectation term with the unconditional expectation, which is zero. Similar considerations motivate us to derive the results in Sections (ref) and (ref) in a simplified setting where $\widehat{Y}^{TS}_{i,T}=Y_{i,T}$ and the weights are based on information up to time $T-1$, which guarantees independence.
In this section we focus on forecasting and show conditions under which IW represents the optimal decision rule at the individual level in terms of two criteria: MSFE or Minimax Regret. In Appendix (ref), we show the analogous results for estimation.
The model is:
where $A_i \sim (\mu, \lambda_i^2)$ and $U_{i,t} \sim (0, \sigma_i^2)$. Here, $A_i, U_{i,1},\ldots,U_{i,T}$ are random variables, whereas $\mu$, $\lambda_i^2$ and $\sigma_i^2$ are parameters. In other words, we take the frequentist approach.
This section considers a simplified setting for IW where the time series forecast is the time-$T$ outcome instead of the time series mean and the IW weights are based on data up to time $T-1$ instead of $T$. These assumptions imply that the weights and the time series forecasts are independent, which makes the theoretical results more transparent and intuitive. In this section we thus consider the following IW forecast:
where ${W}_{i,T-1}$ depends on time series data up to time $T-1$. We make the following assumptions.
Henceforth, we focus on model ((ref)), with the understanding that $Y_{i,t}$ are either raw outcomes or residuals such as ($\ref{panelres})$ or ($\ref{vamres}$) (in a large-$N$ setting).
This section discusses the two criteria that we consider for evaluating the performance of IW: MSFE and Minimax Regret. Consider a situation where there is uncertainty about the parameter $\theta_i = (\lambda^2_i,\sigma^2_i)$. The MSFE of forecast $m \in \mathcal{M}$ for a given $\theta_i$ is
\[ \mathrm{MSFE}(m, \theta_i)=\mathbb{E} \left[ \left( Y_{i,T+1} - \widehat{Y}^{m}_{i,T} \right)^2 \right]. \]
The next lemma derives the MSFEs of TS, Pool and IW in ((ref)).
Lemma (ref) suggests that the trade-off between TS and Pool in terms of MSFE depends on the “signal-to-noise" ratio $\lambda_i^2/\sigma_i^2$: Pool dominates when the ratio is less than 1 and TS dominates when it is greater than 1. Knowing the parameters would allow one to choose the best forecast for individual $i$; however in the presence of uncertainty about the parameters it is not possible to choose a forecast optimally. We thus pursue an alternative route. We seek a robust rule that performs well over the entire parameter space, in the sense of avoiding large errors when TS and Pool have different accuracy and improving on both TS and Pool when they have similar accuracy. The following sections show that IW can accomplish both goals.
We first formalize the notion of robustness that we consider here, based on the Minimax Regret criterion. Let $\mathcal{M}$ include $\mathrm{TS}$, $\mathrm{Pool}$, and $\mathrm{IW}$. We define regret as
The Minimax Regret (MMR) criterion selects the forecast $m$ that minimizes the maximum regret \[ \max_{\theta_i \in \Theta} R(m,\theta_i), \] where $\Theta$ is the parameter space. The form of regret here is similar to that of regret in decision theory without sample data (e.g., see equation (3) in manski2019econometrics). The MMR criterion is championed by manski2019econometrics.\footnote{See Section A.2 in manski2019econometrics and references therein for a detailed discussion.} The regret in (ref) is defined relative to the best forecast (in terms of MSFE) out of a set of three because the goal in this section is to choose among IW, TS, and Pool.\footnote{If we had adopted the criterion of Minimax instead of Minimax Regret, we would have an obvious but trivial solution that TS is preferred to Pool if $\max_{i} \sigma_i^2 < \max_{i} \lambda_i^2$ and vice versa. In this setting, it is not necessarily the case that IW provides a better performance in terms of minimizing the maximum MSFE.}
In this section, we show the conditions under which IW is optimal in terms of Minimax Regret. We restrict our attention to the parameter space represented in Figure 1 below, where the signal-to-noise ratio $\lambda_i^2/\sigma_i^2$ ranges from $1-\nu$ to $1+\nu$ for some $0 \leq \nu < 1$.
Considering a neighbourhood of 1 is natural, as we saw that this point represents the case where TS and Pool are equally accurate. The radius of the neighbourhood is constrained by the fact that the signal-to-noise ratio cannot be negative, so in practice we only exclude cases where TS strongly dominates, due to large variance of the RE and low variance of the error.\footnote{The adoption of a common $\nu$ for the upper and lower bounds is only for convenience in deriving analytical results. Figures (ref) and (ref) below make it clear that we are being conservative, as increasing the upper bound on $\lambda_i^2/\sigma_i^2$ would not change the conclusions of the Minimax Regret analysis.}
The next theorem shows that IW (uniquely) minimizes maximum regret among TS, Pool, and IW under Assumptions (ref) and (ref).
The theorem shows when the improvement of IW over TS and Pool in terms of regret is strict. For example, any constant weight between 0 and 1 provides an improvement. Furthermore, keeping all else equal, a weight that is a genuine function of the RE that satisfies Assumption (ref) with a strict inequality will deliver an additional improvement in performance. Existing Bayesian shrinkage approaches such as the JS estimator, for example, deliver weights that are strictly between 0 and 1 but that do not depend on $A_i$. This means that JS outperforms (in terms of Minimax Regret) using TS or Pool for all individuals, but it can be, in turn, outperformed by any weight satisfying Assumption (ref) that is a genuine function of the RE. The theorem thus illustrates the potential benefits of individual weights that are based on information in the time series dimension, and thus capture the RE, relative to existing shrinkage approaches that leverage instead the cross-sectional dimension.
We illustrate the findings of Theorem (ref) in Figures (ref) and (ref). Consider one individual (so drop the subscript $i$) observed over 4 time periods, with $U_{1},...,U_{4}$ drawn independently from a $\mathcal{N}(0, 1)$ and $A$ drawn from a $\mathcal{N}(0, \lambda^2)$. Repeating the simulation a large number of times allows us to approximate the individual MSFE and regret when forecasting $Y_4$ at time $T=3$ using either TS, Pool or IW. The figures plot these MSFEs and regrets as a function of the signal-to-noise ratio. For IW, we consider the feasible Minimax Regret optimal rule (IW-MR) that we derive in equations ((ref)) and ((ref)) below.\footnote{ Specifically, we have $\widehat{Y}^{TS}_{3}=Y_3, \widehat{Y}^{Pool}_{3}=0$ and $\widehat{Y}^{IW-MR}_{3}=Y_{3} {W}_{2}$, with $W_2= 1 - 1/\sqrt{\frac{ \max \{ Y_{1}^2,Y_{2}^2 \}}{0.5(Y_{1} - Y_{2})^2 } + 1}$.} Figure (ref) shows that no forecast uniformly dominates in terms of MSFE over the parameter space; however, IW is the most accurate over the majority of the parameter space, except for very small values of the signal-to-noise ratio, when Pool dominates. Figure (ref) shows that IW is additionally Minimax Regret optimal. To see why, note that the regret for TS (dashed line) achieves a maximum value of around 1 when the signal-to-noise ratio is close to zero. The regret for Pool (dotted line) obtains a maximum value of around 1.4 when the signal-to-noise ratio is large. The regret for IW (solid line) achieves a maximum value of around 0.27, when the signal-to-noise ratio is close to zero. Thus, over the parameter space, the Minimax Regret optimal rule is IW, since it has the smallest maximum regret among the three rules.
Figure (ref) suggests that IW outperforms TS and Pool in terms of MSFE when the signal-to-noise ratio is in a neighborhood of 1. One may wonder if this result holds regardless of the data-generating process. The following theorem shows that, indeed, IW outperforms TS and Pool in terms of MSFE regardless of the data-generating process, if we restrict attention to the case where the signal-to-noise ratio equals 1. This means that IW is not only robust - i.e. optimal in terms of Minimax Regret - but it is also optimal in terms of MSFE when TS and Pool are equally accurate and thus would be indistinguishable.
The theorem shows that IW is weakly more accurate than TS and Pool when the two forecasts have equal accuracy. Furthermore, as in the case of the Minimax Regret optimality results in Theorem (ref), a strict accuracy improvement can be obtained when the weights are strictly between 0 and 1 or are genuine functions of the RE. This shows that considering individual weights that leverage time series information to capture the RE can deliver a strict improvement in accuracy in terms of MSFE.
In this section we perform a simulation exercise to illustrate the result of Theorem (ref) and further show how the accuracy gains of IW are linked to the heaviness in the tails of the RE distribution. As for Figures (ref) and (ref), we consider one individual observed over 4 time periods. We now however focus on the case $\sigma^2=\lambda^2=1$ (making TS and Pool equally accurate) with $U_{1},...,U_{4}$ drawn independently from a $\mathcal{N}(0, 1)$ and $A$ drawn from a Pareto distribution with different degrees of tail heaviness (in a way that ensures that $\sigma^2=\lambda^2$).\footnote{We consider the double Pareto distribution with pdf $f(x; \theta; \beta)=\theta/(2\beta)
$ with the following parameter combinations for the shape ($\theta$) and scale ($\beta$) parameters: $(2.3, .5)$, $(3,1)$, $(5, 2.45)$, $(50, 34.5)$. Note that population moments of order $\theta$ or greater do not exist. We thus quantify the tail heaviness of the distribution of $A_i$ by reporting a robust quantile-based measure of kurtosis, the Crow-Siddiqui measure ($CS= (Q_{0.975}-Q_{0.025})/(Q_{0.75}-Q_{0.25})$). Tor each of the above four combinations, this respectively equals 7.58, 6.59, 5.51, 4.42.} Repeating the simulation a large number of times allows us to approximate the individual MSFE when forecasting $Y_4$ at time $T=3$ using either TS, Pool or IW. For IW we consider the feasible rule IW-MR in footnote \ref{footIWMR}. Figure \ref{fig:mmr_equallyaccurate} plots the MSFEs of TS, Pool and IW as a function of the heaviness in the tails of the distribution of $A$, as captured by the Crow-Siddiqui measure of kurtosis (on the x-axis). The figure shows that IW improves on the performance of TS and Pool when TS and Pool are equally accurate, confirming the findings of Theorem (ref). Furthermore, the accuracy gains of IW are larger the heavier the tails of the RE.
It is easy to verify that all the feasible weights reported in Section (ref) satisfy Assumption (ref) with a strict inequality. For all weights, the term $(A_i-\mu)^2$ appears in the denominator of $1-W_{i,T}$, whereas the remaining terms only depend on the $U_{i,t}'s$. This implies that large values of $(A_i-\mu)^2$ are associated with small values of the weight on the pooled forecast $\mu$. To illustrate how Assumption (ref) is linked to tail heaviness, we consider the same simulation design as that obtained to produce Figure (ref) and compute the covariance in Assumption (ref) (focusing for simplicity on IW-MR only). The four distributions have increasing tail heaviness, while everything else that would otherwise affect the weights is kept fixed. We find that this covariance becomes more negative as the tail heaviness increases (it respectively equals -0.134, -0.168, -0.216, -0.254 for the four levels of tail heaviness). This implies that heavy tails make the inequality in Assumption (ref) more pronounced, which, as shown by Theorems (ref) and (ref), translates into larger gains of IW relative to TS and Pool.
The findings in the previous sections show the benefits of IW in terms of individual performance, but they also have implications for aggregate performance. Figure (ref), in particular, provides some intuition for how the distribution of the signal-to-noise ratio $\lambda^2_i/\sigma^2_i$ across $i$ (which we do not restrict in any way) affects aggregate performance as measured by the average MSFE. If there are enough individuals for which the signal-to-noise ratio is in the range where IW dominates in terms of individual MSFE, IW will dominate also in terms of average MSFE. In addition, our simulations below will show how the tail properties of the distribution of RE can be linked to improved performance of IW relative to shrinkage estimators. Another simulation will show that IW can be beneficial not only for outliers, but also for individuals that are near the mean of the distribution. This implies that improvements in terms of aggregate performance of IW relative to existing methods can be linked to how many individuals fall in the tails and/or near the mean of the RE distribution.
The results in the previous section show that individual weights are optimal under assumption (ref), but do not directly provide a way to derive feasible weights. In this section, we show that we can derive feasible weights that satisfy this assumption. Here we describe three types of weights. For the first two, as in the previous section, we focus on the simplified setting in section (ref) for illustrative ease, but, again, the weights that we propose to use in practice and that we consider in the simulations and empirical application are the general weights reported in section (ref) and derived in Appendix C under general conditions. The third type of weights are not based on the model assumptions and thus are the same as in the general case.
The first set of feasible weights are based on the oracle weights that minimize the individual MSFE, $$\mathrm{MSFE}(\widehat{Y}^{IW}_{i,T} )=\mathbb{E} \left[ \left( Y_{i,T+1} - \widehat{Y}^{IW}_{i,T} \right)^2 \right],$$ which are functions of the individual variance parameters\footnote{These oracle weights follow for example from equation (9) in Chapter 4 of timmermann2006forecast, using the fact that the joint distribution of $Y_{i,T+1}$ and ${Y}_{i,T}$ is $$
\sim \left(
,
\right),$$ which gives the optimal weight on ${Y}_{i,T}$ as the product between the inverse of the variance of the forecast and the covariance between the outcome and the forecast. The linear combination with $W^{o}_{i}$ as weight could also be obtained by applying the ``best linear rule" in equation (9.4), page 129 of \cite{efron1973stein}, to ${Y}_{i,T}|A_i\sim(A_i,\sigma_i^2)$ and $A_i\sim(0,\lambda_i^2)$.}:
Estimated oracle weights at time $T-1$ can be obtained as
These weights use the fact that: $\Sigma^{T-1}_{t=1} (Y_{i,t}-\mu)^2 /(T-1)$ is an unbiased estimator of $\lambda_i^2+\sigma_i^2$ and that $\widehat{\sigma}^2_i=\Sigma^{T-2}_{t=1} (Y_{i,t} - Y_{i,t+1})^2 /2(T-2)$ is an unbiased estimator of $\sigma_i^2$.\footnote{To see that $\widehat{\sigma}^2_i$ is an unbiased estimator of $\sigma_i^2$ note that:
}
The short time dimension leads to imprecise estimates of these parameters, which can negatively impact the performance of feasible oracle weights. Furthermore, the subtraction in the numerator may produce negative weights. While taking the positive part - as we do for the general weights reported in Section (ref) and derived in Appendix C - partially alleviates this issue, our simulations indicate that these weights still perform poorly in practice.
These considerations motivate our focus on developing feasible weights that are robust, specifically those that optimize Minimax Regret across the unknown parameter space.
In order to obtain feasible Minimax Regret optimal weights we shift from unconditional MSFE to MSFE that is conditional on the information set at time $T-1$. The following lemma is the analog of Lemma (ref) for the conditional MSFE.
It is easy to verify that the weights that minimize $\mathrm{MSFE}(\mathrm{IW}, \theta_i | Y_{i,1},...,Y_{i,T-1})$ are given by:
In this section, we consider a different type of regret, defined as the difference between the conditional MSFE for a generic weight $W_{i,T-1}$ and the conditional MSFE corresponding to the conditionally optimal weights $W^*_{i,T-1}$ in (ref):
where
The form of regret in (ref) is similar to that of regret in statistical decision theory (e.g., see equation (6) in manski2019econometrics).
The following theorem derives the optimal minimax regret weights under the assumption that we can bound the random variable $\zeta_{i,T-1}^2$.
In practice, the value of the bound $\tilde{\zeta}_{i,T-1}^2$ is uncertain, but the following heuristic rule can be used to obtain feasible weights. Assuming $T \geq 3$, we define
where $\mu$ is either known or approximated by the pooled mean. Intuitively, the denominator \[ \sum_{t=1}^{T-2} (Y_{i,t} - Y_{i,t+1})^2 / [2(T-2)] \] is an unbiased estimator of $\sigma_i^2$, the denominator of $\zeta_{i,T-1}^2$. The numerator serves as a proxy for an upper bound on $\kappa_{i,T-1}^2$, the numerator of $\zeta_{i,T-1}^2$.
Although the construction is heuristic, we may interpret our result as a minimax-regret optimal rule conditional on $\tilde{\zeta}_{i,T-1}^2 = \widehat{\tilde{\zeta}_{i,T-1}^2}$. This is similar in spirit to partial identification settings where the outcome variable is known to lie within a bounded interval (e.g., $Y \in [y_{\min}, y_{\max}]$). When $y_{\min}$ and $y_{\max}$ are unknown, it is common to use the sample minimum and maximum as proxies, and interpret the resulting identification region as conditional on these sample bounds.
The weights we derive in this section do not rely on the model and the assumptions, and are thus applicable in more general settings. The weights are based on comparing the (in-sample or out-of-sample) MSFE at time $T$ of the TS and Pool forecasts. These weights are analogous to those considered in the time-series forecast combination literature (e.g., bates1969combination, stock1998comparison), with the difference that the MSFE is computed here for each individual over a very small time series sample (possibly containing only one observation). As in stock1998comparison, the weights ignore any correlation between the TS and the Pool forecasts.\footnote{In the time series literature, these weights are known to perform well even when the time dimension is large because of the challenges in estimating correlations precisely. See, e.g., the discussion in stock1998comparison.}
The in-sample inverse MSFE weights are given by:
The out-of-sample inverse MSFE weights are given by:
Note that here we base ${W}_{i,T}^{OOS} $ only on the out-of-sample forecast errors at time $T$ corresponding to the TS and Pool forecasts computed on the sample up to time $T-1$. Depending on the magnitude of $T$, one could also compute the out-of-sample MSFEs using more than just one out-of-sample period. For example, one could select $P<T$ and consider
Finally, we note that one could consider “rolling-window" forecasts, both as the original TS and Pool forecasts and in the computation of the weights. In this case, both TS and Pool forecasts at time $t$ would be based only on the $R<t$ most recent observations, rather than all available observations up to time $t$.
In this section we first study the finite sample performance of alternative feasible IW weights. We then compare IW to JS.
We consider one individual (so we here drop the subscript $i$) observed over 3 time periods, with errors $U_{1},...,U_{3}$ drawn independently from a $\mathcal{N}(0, 1)$ and RE $A$ drawn from a $\mathcal{N}(0,\lambda^2)$, with $\lambda^2$ taking $50$ equally-spaced values on the grid $[0.001, 2]$. Repeating the simulation 10000 times allows us to approximate the individual MSFE when forecasting $Y_3$ at time $T=2$ using the different IW feasible weights reported in Section (ref) (with $\widehat{Y}_{i,T}^{TS}=\bar{Y}_i$ and $\mu=0$). Figures (ref) and (ref) respectively report the MSFEs of each IW rule, divided by the MSFE of IW-MR, and the Regret of each IW rule as a function of the signal-to-noise ratio (which here equals $\lambda^2$). In Figure (ref), a line above 1 means that the rule is dominated by IW-MR.
Figures (ref) and (ref) illustrate the dominance of IW-MR (black solid line) over the other feasible rules, in terms of both MSFE and Regret. Figure (ref) shows that IW-MR is Minimax Regret optimal over the parameter space. IW-MSFE-IS (blue dotted line) is uniformly dominated in terms of both criteria by IW-MR, although by a small amount. The performance of IW-MSFE-OOS (red dashed line) and IW-O (green dashed-dotted line) depends on the signal-to-noise ratio, both outperforming IW-MR when the signal-to-noise ratio is very low, but performing poorly over the rest of the parameter space. We thus conclude that IW-MR is the preferred rule, closely followed by IW-MSFE-IS.
In this section we compare the performance of IW-MR to that of the JS forecast (with estimated weights, as reported after equation ((ref))). The draws in the following simulations can be interpreted in two different ways. First, they can be seen as different possible draws of the RE for one individual. Second, they can be seen as draws for different individuals that have the same distribution of the RE. Averages across simulations accordingly have a different interpretation: they approximate the individual MSFE in the first interpretation and the aggregate MSFE in the second. The JS forecast in the first interpretation is simply an IW with constant weights that do not depend on the RE (see the discussion after Theorem (ref)). In the second interpretation, JS is the forecast that exploits information from the cross-sectional dimension, in contrast to IW, which leverages the time series dimension. Note that the assumptions of parameter homogeneity made by JS are satisfied in all the designs we consider.
We start by visually illustrating how IW overcomes the tyranny of the majority phenomenon that affects JS. Henceforth, we focus on the IW-MR rule, which we saw in the previous section generally outperforms the other feasible rules. We consider 10000 simulations of outcomes generated as $Y_{t} = A + U_{t}$, with $t=1,...,3$, $U_{t} \sim \mathcal{N}(0,1)$, independent across $t$. For the RE we consider the following designs:
These designs correspond to an increasing heaviness in the tails of the RE distribution.
We compare IW-MR as described in Section (ref) (with $\widehat{Y}^{Pool}_T=0$) to JS (with estimated weights, as reported after equation ((ref))).
Figures (ref) - (ref) report the difference $\Delta SFE$ between the squared forecast errors of forecasts made at $T=2$ for IW-MR versus JS, for the different designs. The horizontal axis reports the value of $A$. The figures illustrate the tyranny of the majority phenomenon: JS tends to make larger errors than IW-MR (the dots fall below zero) for RE in the tails and also near the center of the distribution. This pattern is not yet visible in Figure (ref) for the normal design with low variance, where the cloud appears symmetric relative to the horizontal axis, but it is clear in the remaining figures. For example, in Figure (ref) (the normal design with larger variance) the cloud is heart-shaped, showing the superior performance of IW-MR near the center of the distribution. Figure (ref) (the Laplace design) also shows the heart shape but also the superior performance of IW in the tails. The improvement in the tails is starkly evident in Figure (ref) (the Double Pareto design), where the cloud has an inverted U-shape. These results illustrate that what matters for the tyranny of the majority is not only the tail heaviness of the RE distribution, but its relationship to the variance: the design in Figure (ref) shows that the phenomenon is present even when the distribution has thin tails, but large variance. This is intuitive, as both a high variance and heavy tails make it worthwhile to link the shrinkage to the RE (IW) instead of shrinking every individual by the same amount (JS).
The second interpretation of the simulation discussed above, which views the different draws as different individuals, allows us to also analyze aggregate performance. Averaging the $\Delta SFE$'s reported in each figure across $i$ gives a measure of the relative aggregate performance of IW-MR and JS. These average equal 0.019, 0.025, -0.005 and -0.027, respectively in Figures (ref) - (ref). The relative aggregate performance thus depends on the tail properties of the distribution of RE, with JS dominating in the normal cases and IW-MR dominating in the heavy-tailed cases.
We consider two applications of IW (specifically, IW-MR from Section (ref)).
In this section we use IW to extend the analysis in kline2022systemic, assessing the extent to which large U.S. employers systemically discriminate job applicants based on gender. We compare the performance of IW-MR to that of JS (with estimated weights, as reported after equation ((ref))) and EB (specifically, the deconvolution estimator of efron2016empirical, henceforth Efron).
We use the panel dataset in kline2022systemic on an experiment that consisted of sending fictitious applications to jobs posted by 108 of the largest U.S. employers. For each firm, 125 entry-level vacancies were sampled and, for each vacancy, 8 job applications with random characteristics were sent to the employer. Sampling was organized in 5 waves (between October 2019 and April 2021). Focusing on firms sampled in all waves yields a balanced panel of $N=72$ firms over $T=5$ waves.\footnote{Accounting for vacancy closures and the exclusion of some firms from some waves reduced the number of applications to $65,400$.}
Applications were sent in pairs, one randomly assigned a distinctively female name and the other a distinctively male name. For details on the other observables see kline2022systemic. The primary outcome in kline2022systemic is whether the employer attempted to contact the applicant within 30 days of applying. The gender contact gap is defined as the firm-level difference between the contact rate (the ratio of number of contacts and number of received applications) for male and that for female applications.
The results in kline2022systemic are based on Efron. The approach considers firm-specific studentized contact gaps, $y_{i.t}=Y_{i,t}/s_i$, where $Y_{i,t}$ is the contact gap and $s_i$ is the standard deviation of contact gaps across different job applications for firm $i$. These are modelled as
$$y_{i,t}=a_i+u_{i,t}, \;\;\;\;\; u_{i,t}\sim \mathcal{N}(0,1) \;\;\;\;\; a_i \sim G_{a}, \;\;\;\;\; \text{for} \; i=1, ..., 72. $$ $G_{a}$ is assumed to belong to an exponential family, parameterized by a fifth-order spline. By pooling observations from all five waves, Efron yields penalized Maximum Likelihood estimates of the spline parameters and thus an implied distribution $\hat{G}_{a}$ of studentized contact gaps with density $\hat{g}_{a}= d \hat{G}_{a}$. One can then recover the distribution $\hat{G}_{A}$ of the RE for the unstudentized contact gaps $Y_{it}$ under the assumption of independence between the RE and $s_i$. In particular, the density $\hat{g}_{A} = d \hat{G}_{A}$ at each point $x$ is obtained as $\hat{g}_{A}(x)= \frac{1}{N}\Sigma^N_{i=1} \frac{1}{s_i} \hat{g}_{a}(\frac{x}{s_i})$. \footnote{To perform the deconvolution, the choice of two tuning parameters is required: the order of the spline and the penalty parameter of the first-step maximum likelihood procedure. The latter is optimally calibrated to obtain a variance matching the bias-corrected estimate in Table IV of kline2022systemic.}
We investigate whether the differences between IW and Efron matter in terms of estimation and policy implications. For instance, suppose a counselor's goal is to guide applicants on whether to avoid sending their applications to firms that are identifies as highly discriminatory or discriminatory-- those exceeding a specified contact gap threshold, such as 0.05 or 0 respectively. We thus calculate $\widehat{\text{Prob}}\left(\hat{Y}^k_{i,T}>0.05\right)$ and $\widehat{\text{Prob}}\left(\hat{Y}^k_{i,T}>0\right)$ at $T=5$, for $k \in \{\text{Efron, IW-MR}\}$.\footnote{These probabilities are calculated producing 1-period ahead forecasts at $T=5$ of contact gaps based on in-sample data from all five waves.} We find that the probability of classifying a firm as highly discriminatory (discriminatory) is 4.29% (64.29%) for IW compared to 1.43% (60%) for Efron, suggesting a higher degree of discrimination and thus different policy implications when using IW instead of Efron.
We compare the forecasting performance of IW relative to that of Efron and JS. For each wave $T= 3, 4$, we produce one-step-ahead forecasts of (unstudentized) contact gaps for each firm by the following methods: TS, which uses the time-series mean of contact gaps at time $T$; Pool, which uses the pooled mean at time $T$ and IW-MR. For Efron, we obtain forecasts as posterior mean estimates of the RE.\footnote{We use the code provided by kline2022systemic that produces Figure A13 in their paper, where they assess the out-of-sample forecast accuracy of the posterior means. We adapt the code to use data from waves $1, .., T$ to produce the forecast at $T= 3, 4$.} We then compare the out-of-sample forecasts from each method $k$, $\{\hat{Y}^k_{i,T}\}$ to the actual realizations $\{{Y}_{i,T+1}\}$, for the waves $4,5$. For each forecasting method $k$ and each firm $i$, the MSFE over the out-of-sample period is
Figure (ref) reports the difference $\Delta MSFE$ between the MSFE of forecasts for IW-MR and those for Efron for each firm. The horizontal axis shows the value of the gender contact gap at $T=4$. The figure reveals that Efron tends to make larger errors (the dots fall below zero) for firms that at the time of forecasting fell in the right tail or near the center of the distribution, illustrating a possible tyranny of the majority phenomenon (which could also be due to misspecification of the normality assumption and/or to the effect of the data-dependent choice of regularization parameter used by kline2022systemic when implementing Efron).
We also evaluate the aggregate performance of the different methods by reporting the average MSFE across $i$ in Table (ref), which shows that IW-MR is the best method.
To illustrate the robustness of IW vs. Efron, we conduct a subsampling exercise. We randomly draw $B=1000$ subsamples without replacement, each subsample $b$ consisting of $n_b=20$ firms. For each method $k$, where $k \in \{\text{Efron, IW-MR}\}$, and for each subsample $b$, we calculate the aggregate out-of-sample RMSFE: $$\text{RMSFE}_{b, k} = \sqrt{\frac{1}{n_b} \sum_{i=1}^{n_b} \left(Y_{i, T+1}-\hat{Y}^k_{i,T}\right)^2} $$ at $T=4$, with $\hat{Y}^\text{Efron}_{i,T}$ obtained using the full sample of firms. In Table (ref), we report the minimum, maximum, mean, median and 90th percentile of $\text{RMSFE}_{b, k}$ across the $B=1000$ samples for each method.
The results in Table (ref) demonstrate a sizeable reduction in the worst-case performance under IW compared to Efron. While the aggregate performance (i.e., the Mean or Median columns in the table) is slightly better for IW but comparable between the two methods, the difference in the 90% percentile and Max RMSFE is noticeable: the Max RMSFE is .09312 for Efron versus .08723 for IW, representing a 6.33% improvement. This indicates that Efron is more prone to poor performance depending on the composition of the subsample, whereas IW effectively reduces the risk of large RMSFE values. These findings highlight the robustness of our approach, as evidenced by its consistently strong performance across the state-space simulated through this subsampling exercise.
This section considers an out-of-sample exercise that applies IW to forecasting earnings residuals using the Panel Study of Income Dynamics (PSID).
We consider earnings data from the PSID for 1968-1993.\footnote{We use data up to 1993 because from 1994 a major revision of the survey disrupted the continuity of PSID files, see kim2000notes. Moreover, after 1997 the PSID switched from an annual to a biannual data collection.} We follow the literature on income dynamics (e.g., meghir2004income) and select a sample of male workers, heads of household, aged between 24 and 55 (inclusive). We drop individuals identifying as Latino, with a spell of self-employment, with zero or top-coded wages and with missing records on race and education. We also require that the change in log earnings is not greater than $+5$ or less than $-3$. We consider earnings residuals obtained from a first stage panel data regression of log labor income of an individual $i$ at time $t$, $\tilde{Y}_{i,t}$, on education, a quadratic polynomial in age, race and year dummies. We denote by $Y_{i,t}$ the residuals from this regression.
The goal is to obtain individual one-year-ahead forecasts of earnings residuals $Y_{i,t}$.\footnote{Forecasting earnings residuals is of interest since they measure individual income risk. For instance, accurate forecasting of individual earnings residuals might be useful for prospective lenders when deciding on loan applications.}
We compare the out-of-sample aggregate performance of IW-MR from Section (ref), versus using TS or Pool for all individuals.
We report results for the balanced samples consisting of $N=164$ ($N=790$) individuals with continuous earnings in all consecutive years for 1968-1993 (1968-1980). We further consider an unbalanced sample built using rolling windows of $T=3$ time periods of balanced samples of individuals (which delivers sample sizes ranging from $3960$ to $7912$). Forecasts are based on the model:
We use rolling windows of $T=2$ time periods and compare the out-of-sample forecasts from each method $k$, $\hat{Y}^k_{i,T}$, where $k \in \{\text{TS, Pool, IW-MR}\}$ to the actual realizations ${Y}_{i,T+1}$, for $t=1972,...,1992, i=1,...,N$.
For each forecasting method $k$ and each individual $i$, the MSFE over the out-of-sample period is
Table (ref) reports averages of $MSFE(k,i)$ across $i$ for each forecasting method $k$. The Table shows that, while TS clearly outperforms Pool in terms of average MSFE, IW further improves aggregate accuracy.
To gain some insight into which individuals benefit more from borrowing strength (i.e., are given higher weights to Pool by IW), in Figure (ref) we divide the N=164 individuals of the balanced sample into ten quantiles according to their lagged earnings (the vertical axis) for each year (the horizontal axis). Within each quantile we compute the forecasts given the higher weights by IW-MR: the size of the dots is proportional to the average weight attributed by IW-MR to Pool across individuals in that year and in that quantile.\footnote{We set the size option of the R package ggplot equal to the mean of weights attributed to Pool by the IW-MR rule for each quantile.} Figure (ref) shows that individuals near the median of the earnings residuals distribution are those who benefit from larger weights to Pool.
One possible interpretation of our findings is that in the PSID there is enough unobserved heterogeneity to make the TS forecast outperform Pool in the aggregate (as indicated by Table (ref)). However, an additional improvement in aggregate accuracy can be obtained by using IW, which tends to borrow strength for individuals near the median of the distribution. This finding confirms the usefulness of IW even in terms of aggregate performance.
Estimating random effects and forecasting with micropanels is challenging due to the short time dimension, and existing solutions have shortcomings. In this paper, we introduce a complementary approach that effectively addresses these limitations while imposing minimal assumptions. In practice, our method involves shrinking the time series mean towards the panel mean, utilizing individual-specific weights calculated solely from time series data. We propose three types of feasible weights: estimated oracle weights, Minimax Regret optimal weights, and inverse mean squared forecast error (MSFE) weights. Our findings indicate that Minimax Regret optimal weights offer superior performance, closely followed by (in-sample) inverse MSFE weights.
Our method is applicable to linear panel data models and value-added models, subject to the assumption that covariates have homogeneous coefficients. The extension to models with heterogeneous coefficients for covariates, as well as to alternative loss functions, is straightforward for the inverse-MSFE weights, which directly accommodate more general models and loss functions. Extending the derivation of Minimax Regret optimal weights to more general contexts is inherently more complex, and we reserve this exploration for future research. A further natural extension is to consider more general classes of shrinkage estimators, such as combinations of our individual-level shrinkage estimator and the James Stein estimator.
\nocite{*}