Robust Forecasting

-0.5 in Robust Forecasting

JEL CLASSIFICATION: C11, C14, C23, C53

KEY\ WORDS: Statistical Decision Theory, Dynamic Discrete Choice, Forecasting, Identification, Minimax Loss, Minimax Regret, Panel Data Models, Robustness, Structural Breaks.

\thispagestyle{empty} \setcounter{page}{0}

Introduction

In this paper, we study the problem of forecasting discrete outcomes when the researcher is unable to discriminate among a set of plausible forecast distributions. There are several reasons why the forecaster might face uncertainty about the forecast distribution. A leading case is partial identification, in which the data up to the forecast origin only set-identify a subset of parameters of the forecasting model. Uncertainty about the forecast distribution can also arise when the forecaster expands the set of models to accommodate concerns about model misspecification or structural breaks between the in-sample and forecast period.

Suppose that a subset of parameters of a forecasting model are only set-identified. Should the lack of point identification be a concern for the forecaster? At first glance the answer appears to be “no”: if the parameters in the identified set generate different forecasts and some of these forecasts are less accurate than others, then we should be able to discriminate among the parameters based on the observed data. To the extent that we are unable to do so, the parameterizations should be observationally equivalent and therefore generate the same forecasts. This intuition is confirmed in the context of vector autoregressions (VARs): while the structural form of the VAR may only be set-identified, forecasts only utilize the reduced form of the VAR which is directly identifiable. This intuition is also confirmed in the context of dynamic linear factor models. The parameters are only identified up to a particular normalization of the latent factors, but each normalization leads to identical forecasts. However, the intuition is wrong in many other important settings.

Our paper makes several contributions. First, we show that the VAR intuition does not apply when forecasting using dynamic discrete choice models for panel data. As is well known HonoreTamer2006,Chamberlain2010,Chernozhukovetal2013, the homogeneous parameters and the correlated random effects distribution are set-identified when no parametric assumptions are made about the random effects distribution. We demonstrate that in the panel dynamic discrete choice setting different parameters in the identified set lead to different forecasts, some more accurate than others. Unlike a VAR, the panel dynamic discrete choice model has a non-Markovian structure due to the sequential learning about heterogeneous coefficients. As a consequence, parameterizations that are indistinguishable based on a panel of length $T$ may become distinguishable in a panel of length $T+1$.

Second, we construct forecasts that are “robust” to uncertainty about the parameterization $\theta$ of the forecasting model among a set of parameterizations $\Theta_0$ that are observationally equivalent at the forecast origin $T$. We refer to this as uncertainty about the forecast distribution for short. Our robust forecasts minimize either maximum risk or maximum regret (i.e. risk relative to the infeasible Bayes decision under the true forecast distribution) over the set of forecast distributions. We show that for binary (or classification) loss, quadratic loss, and logarithmic loss, the optimal binary forecast under either robustness criterion depends on two extremum problems which characterize the smallest and largest conditional probabilities for the outcomes being forecast over the set of forecasting distributions. Similarly, robust forecasts in the multinomial case depend in a natural way on a small number of extremum problems. Further, we show that these extremum problems can be simplified by duality arguments for a broad class of models.

Our robust forecasts are not only applicable in settings in which parameters are set-identified, but also in environments in which the forecaster is concerned about model misspecification or there is a structural break at the forecast origin. The common feature is that the forecasts depend on an unknown parameter $\theta$ which takes values in a set $\Theta_0$. Under misspecification or structural breaks, the set $\Theta_0$ indexes an enlarged class of models representing plausible deviations from a benchmark model.

Third, we derive “efficient robust” forecasts to deal with the problem of first estimating the set $\Theta_0$ prior to making the forecast. To do so, we express $\Theta_0$ as a (set-valued) function of an identifiable reduced-form parameter $P$. In order to develop an optimality theory, we evaluate forecasts by their integrated maximum risk or regret, averaging over both $P$ and the data. Under this criterion, the optimal forecast is what we call the Bayesian robust forecast. It is obtained by minimizing the posterior maximum risk or regret which conditions on the data and averages out $P$ based on its posterior distribution.

Fourth, we develop an asymptotic efficiency theory for forecasting discrete outcomes under uncertainty about the parameterization of the forecast distribution when $\Theta_0$ is estimated. We show that in binary and multinomial discrete forecasting problems, forecasts that are asymptotically equivalent to the Bayesian robust forecasts minimize asymptotic integrated maximum risk or regret. We refer to such forecasts as asymptotically efficient-robust. We demonstrate that forecasts obtained by replacing $P$ with an efficient first-stage estimator can be strictly dominated by the Bayesian robust forecast. This suboptimality of plug-in forecasts arises in settings in which key statistics that determine the robust forecast are only directionally, but not fully, differentiable with respect to the identifiable reduced-form parameters. Bagged predictors (see Breiman1996) that replace posterior averaging with averaging across the bootstrap distribution of an efficient estimator of $P$, on the other hand, tend to be asymptotically efficient-robust.

Our paper is related to several literatures. For forecasting short time-series using panel data see, e.g., Baltagi2008, GuKoenker2014, Liu2016, and LiuMoonSchorfheide2018,LiuMoonSchorfheide2015. Applications of partial identification in nonlinear panel data analysis include HonoreTamer2006 and Chernozhukovetal2013. Much of our paper is devoted to forecasting binary outcomes which has been previously considered by, for instance, ElliottLieli2013, LahiriYang2013, and ElliottTimmermann2016.

There is an extensive literature on statistical decision theory following Wald1950. Closely related to our approach are $\Gamma$-minimax (or $\Gamma$-minimax regret) decisions in robust Bayes analysis Robbins1951,Berger1985. In economics, this approach is also related to the multiple priors framework of GilboaSchmeidler1989 and the robustness literature following HansenSargent2001. For econometric applications, Chamberlain2000,Chamberlain2001 derives minimax decision rules under point identification. Kitagawa2012, GiacominiKitagawa2018, and GiacominiKitagawaUhlig2019 study robust Bayesian analysis under set identification.

HiranoWright2017 study the problem of forecasting continuous outcomes under uncertainty about predictor variables in a weak predictor local asymptotic setting. Despite several differences between their work and ours,\footnote{For instance, uncertainty about the parameterization of the forecasting model is resolved asymptotically in their framework whereas it persists in our setting.} they also find that bagging can reduce asymptotic risk. Discrete forecasting has a similar structure to statistical treatment assignment and our efficiency results are related to efficiency results in that literature, most notably HiranoPorter2009. In their setting, a welfare contrast is a smooth function of a point-identified, regularly estimable parameter. Their efficient rules are based on plugging-in an efficient estimator of the parameter. In our setting, uncertainty about the forecast distribution can introduce a type of non-smoothness to the robust forecasting problem. In consequence, our efficient robust forecasts differ from plug-in rules.

The remainder of this paper is organized as follows. Section (ref) describes the setup, our objectives, and introduces motivating examples. Sections (ref) and (ref) derive our robust and efficient robust forecasts for binary and multinomial forecasting settings, respectively. Section (ref) also contains an application to panel models for dynamic binary choice. Section (ref) presents the main results on asymptotic efficiency. Appendix (ref) discusses computation for a broad class of models including semiparametric panel data models. Appendix (ref) contains additional results on robust binary forecasts and all proofs are relegated to Appendix (ref).

Setup, Motivating Examples, and Objectives

Setup

The econometrician wishes to forecast a random variable $Y$ taking values in a finite set $\mathcal Y$. The econometrician assumes $Y$ is distributed according to an (unknown) distribution in a family of forecast distributions $\big\{ \mathbb{P}_\theta(Y = y ) : \theta \in \Theta_0 \big\}$, where $\theta \in \Theta$ denotes a vector of parameters and $\Theta_0 \subseteq \Theta$ indexes the set of forecast distributions over which the forecaster seeks robustness. The forecast distributions $\mathbb{P}_\theta(Y = y ) $ may be conditioned on covariates observed by the econometrician when making the forecast, but we suppress this dependence in what follows to simplify notation.

As we discussed in the Introduction, there are several reasons why the forecaster might be uncertain about the forecast distribution. A leading case is partial identification, in which $\Theta_0$ represents the identified set of parameters that are observationally equivalent up to the forecast origin. Uncertainty about the forecast distribution can also arise under concerns about model misspecification or structural breaks. In these settings, $\Theta_0$ indexes an enlarged class of models representing plausible deviations from a benchmark model.

Motivating Examples

To fix ideas and illustrate the broad applicability of our results, we now present a number of examples of where this forecasting problem arises. The first four examples use a panel data model for dynamic discrete choice to show how our approach accommodates concerns about model misspecification and structural breaks in a unified manner, though these are relevant concerns for {\em any} forecasting model. The last two examples involve counterfactuals and treatment assignments.

\paragraph*{Example 1: Semiparametric random effects model for dynamic binary choice.} Let

where $\mathbb{I}[ y \ge a] = 1$ if $y \ge a$ and $0$ otherwise, $Y_i^t = (Y_{i1},...,Y_{it})'$, and $y^t \in \{0,1\}^t$. The econometrician observes $Y_i^T = (Y_{i1},...,Y_{iT})'$ for $i=1,\ldots,n$ where $T$ is fixed. To avoid the initial conditions problem, the econometrician treats $Y_{i0}$ as unobserved and specifies a joint distribution $\Pi_{\lambda,y}$ over $Y_{i0}$ and $\lambda_i$. As is well known, $\beta$ is not point-identified when $T$ is small and no parametric restrictions are placed on $\Pi_{\lambda,y}$ (see, e.g., Cox1958, Chamberlain1985, and Magnac2000). Moreover, $\Pi_{\lambda,y}$ and the $\Phi_t$ are not nonparametrically point-identified for any $T$.

The econometrician wishes to forecast individual-level outcomes $Y_{iT+1}$ conditional upon an individual's history $Y_i^T = y^T \in \{0,1\}^T$. Suppose that the econometrician assumes each of the $\Phi_t$ takes a parametric form $\Phi$, such as logistic or standard normal. The identified set $\Theta_0$ then consists of all $(\Pi_{\lambda,y},\beta)$ for which the model-implied probabilities of observing sequences $Y_i^T = y^T \in \{0,1\}^T$ are equal to the probabilities observed in the data up to date $T$:

where $ p(y^T | \beta, \Pi_{\lambda,y})$ denotes the model-implied probabilities and $p(y^T)$ denotes the true (population) probabilities of observing $Y_i^T = y^T$. In the above notation, $Y = Y_{iT+1}$ and the forecast probability $\mathbb{P}_\theta$ denotes the conditional probability over $Y_{iT+1}$ given $Y_i^T = y^T$:

\paragraph*{Example 2: Misspecification.} Consider the setup described in Example 1, but suppose that the econometrician adopted a parametric correlated random effect model, $\Pi_{\lambda,y} = \Pi(\lambda,y_0; \xi)$ for $\xi \in \Xi$, a set of auxiliary parameters. The econometrician is worried that this parametric random effects specification is misspecified, and so allows for the possibility that $\Pi_{\lambda,y} \in N(\xi)$, a neighborhood of $\Pi(\lambda,y; \xi)$. Suppose the econometrician again sets $\Phi_t = \Phi$ for all $t$. The set $\Theta_0$ is \[ \Theta_0 = \{ \theta = (\beta,\xi,\Pi_{\lambda,y}) \in \Theta : p(y^T | \beta, \Pi_{\lambda,y}) = p(y^T) \;\; \forall \, y^T \in \{0,1\}^T \;\; \mbox{and} \;\; \Pi_{\lambda,y} \in N(\xi) \}\,. \] This setup was considered by BonhommeWeidner under local misspecification, where $N(\xi)$ are Kullback--Leibler neighborhoods $N(\xi) = \big\{ \Pi : K(\Pi \,\|\, \Pi(\,\cdot\,;\xi)) \leq \delta \big\}$ for each $\xi \in \Xi$ with $\delta \downarrow 0$ as $n \to \infty$, so that worst-case misspecification bias and sampling uncertainty are of the same order asymptotically.\footnote{Note that the emphasis of BonhommeWeidner is on estimating posterior average effects whereas we focus on forecasting discrete (e.g. individual-level) outcomes.} We instead treat $\delta > 0$ as fixed, allowing global misspecification. $\Box$

\paragraph*{Example 3: Structural breaks.} Three types of breaks can, in principle, occur at the forecast origin $T$ in Example 1: a break in the distribution of the $U_{it}$, a break in the individual effects $\lambda_i$, and a break in $\beta$. Suppose the econometrician again takes $\Phi_t = \Phi$ for dates $t = 1,\ldots,T$, but allows for the possibility that $\Phi_{T+1} \neq \Phi$. For instance, the econometrician might want to allow for $\Phi_{T+1} \in N$, a neighborhood of $\Phi$. Even if $\beta$ and $\Pi_{\lambda,y}$ were known at date $T$, there would still be a set of forecast distributions for $Y_{iT+1}$ corresponding to different $\Phi_{T+1} \in N$. Using the above notation, we can redefine $\Theta_0$ as \[ \Theta_0 = \big\{ \theta = (\beta,\Pi_{\lambda,y},\Phi_{T+1}) \in \Theta : p(y^T | \beta, \Pi_{\lambda,y}) = p(y^T) \;\; \forall \, y^T \in \{0,1\}^T \;\; \mbox{and} \;\; \Phi_{T+1} \in N \big\}\,, \] and replace $\Phi$ in the definition of $\mathbb{P}_\theta(Y=1)$ in ((ref)) with $\Phi_{T+1}$. Breaks in $\lambda_i$ can be viewed as a location shift of the distribution $\Phi_t$ and are subsumed under breaks in the distribution of $U_{it}$ for suitable choice of $N$. Breaks in $\beta$ can be handled by defining \[ \Theta_0 = \big\{ \theta = (\beta,\beta_{T+1},\Pi_{\lambda,y}) \in \Theta : p(y^T | \beta, \Pi_{\lambda,y}) = p(y^T) \;\; \forall \, y^T \in \{0,1\}^T \;\; \mbox{and} \;\; | \beta - \beta_{T+1}| \leq \delta \big\}\,, \] and by replacing $\Phi(\beta y_{iT} + \lambda) $ in ((ref)) with $\Phi(\beta_{T+1} y_{iT} + \lambda) $. $\Box$

\paragraph*{Example 4: Semiparametric random effects model for dynamic multinomial choice.} Let \[ Y_{it+1} = \arg \max_m \left( U_{it+1}^m + \varepsilon_{it+1}^m \right) \,, \;\; U_{it+1}^0 = 0 \,, \;\; U_{it+1}^m = u_{mt+1}(X_{it},Y_i^{t};\phi,\lambda_i,Y_{i0}) \,, \;\; m = 1,\ldots,M \,, \] where $\varepsilon_{it} = (\varepsilon_{it}^0,\ldots,\varepsilon_{it}^M)'$ is a vector of utility shocks with $\varepsilon_{it}|X_{it},\lambda_i \sim \Phi_t$ for each $t$, $\Phi_t$ is a potentially time-varying distribution, $\phi$ is a vector of homogeneous parameters, $\lambda_i$ is a vector of heterogeneous parameters, and $X_{it}$ is a vector of exogenous regressors. The econometrician observes $Y_i^T = (Y_{i1},\ldots,Y_{iT})'$ and $X_i^T = (X_{i1},\ldots,X_{iT})'$ for $i = 1,\ldots,n$ where $T$ is fixed and $n \to \infty$. To avoid the initial conditions problem, the econometrician specifies a joint distribution $\Pi_{\lambda,y}$ for $(\lambda_i,Y_{i0})$. As with Example 1, identification of model parameters with fixed $T$ and $n \to \infty$ is delicate, especially when parametric assumptions about the $\Phi_t$ and/or $\Pi_{\lambda,y}$ are relaxed; see HonoreKyriazidou2000, Chernozhukovetal2013, KhanOuyangTamer2019, and references therein. Identified sets and forecast probabilities for individual-level outcomes are constructed in a similar manner to Example 1. $\Box$

\paragraph*{Example 5: Counterfactuals in structural models.} Counterfactuals in structural models are also subsumed in our framework when the outcome of interest is discrete, as is often the case for static or dynamic models of discrete choice or discrete games (e.g. firm entry/exit). In the above notation, $\theta$ are the structural parameters estimated under one policy regime, the econometrician wishes to predict a variable $Y$, and the model implies that $Y$ is distributed according to $\mathbb{P}_\theta$ under the intervention. Partial identification can arise on two fronts. First, the model may itself be specified flexibly, leading to a non-singleton identified set $\Theta_0$ of structural parameters. Second, the potential for multiple equilibria and lack of knowledge about an equilibrium selection mechanism under the intervention may lead to a nontrivial set of forecast distributions. This can be subsumed by treating the selection mechanism itself as part of $\theta$, with $\Theta_0$ indexing distributions in a manner that is robust to the type of selection mechanism (see, e.g., Jia2008, CilibertoTamer, and Grieco2014). $\Box$

\paragraph*{Example 6: Treatment assignment.} The problem of making discrete forecasts has a very similar structure to a treatment assignment problem, e.g., determining whether an individual should be vaccinated. Suppose the econometrician has access to a sample of observational data of size $n$ and observes for an individual $i$ the triplet $X_i = \big(D_i, W_{0i}(1-D_i), W_{1i} D_i \big)$, where $D_i$ is a treatment indicator, and $W_0$ and $W_1$ are the potential outcomes (“welfare”) of the untreated and the treated individuals. As the sample size tends to infinity, the econometrician is able to estimate the reduced form expectations $P= \big( \mathbb{E}[D], \mathbb{E}[W_0(1-D)], \mathbb{E}[W_1D] \big)$.

Let $\mathbb{P}_\theta$ denote the joint distribution of $(D,W_{0},W_{1})$. To keep the example simple, we assume that the potential outcomes are binary and take values $\{a_{0}, a_{1}\}$ and $\{b_{0}, b_{1}\}$. Thus, the distribution $F_\theta$ is discrete with support on $\{0,1\} \times \{a_{0}, a_{1}\} \times \{b_{0}, b_{1}\}$. The support points of the potential outcome distribution can be easily point-identified based on two untreated (and two treated) individuals with different outcomes. Thus, we exclude the support points from the definition of $\theta$ and $P$. Using the notation that $\theta_{ijk} = \mathbb{P}\big( D=i, W_0=a_{j}, W_1=b_{k} \big)$, the identified set $\Theta_0(P)$ is defined by the following set of linear restrictions:

where $\theta$ stacks the $\theta_{ijk}$ probabilities.

Define the indicator variable $Y=\mathbb{I}[W_1 \ge W_0]$ which measures whether the treatment effect is (weakly) positive or not. From a policy maker's perspective the treatment effect for an individual not included in the initial trial is uncertain and the treatment decision $d \in \{0,1\}$ can be viewed as a forecast of $Y \in \{ 0,1\}$ with the understanding that the individual should be treated if the point forecast of $Y$ is one and not treated otherwise.

Dehejia2005 analyzed this problem in a decision-theoretic framework under point identification with binary treatments. However, the example highlights the well-known result that the distribution of welfare rankings can be partially identified (see, e.g., Manski1996,Manski2000 and HeckmanSmithClements1997). Manski2000,Manski2002,Manski2004,Manski2007treatment used a decision-theoretic framework to analyze optimal treatment in a planning problem under partial identification and advocated minimax and minimax regret approaches.\footnote{Manski2002 focuses on population welfare objective whereas here we focus on individual-level outcomes. See also ManskiTetenov2007, HiranoPorter2009, Tetenov2012, and KitagawaTetenov2018 amongst others, for the analysis of treatment rules under social welfare objectives.} $\Box$

Objectives

We derive two types of forecasts that deal with uncertainty about the forecast distribution. The first are robust forecasts which seek to robustify the forecast with respect to $Y$ being distributed according to any distribution in the class $\{\mathbb{P}_\theta : \theta \in \Theta_0\}$. We use minimax and minimax regret criteria as our notion of robustness. The second are efficient robust forecasts which deal with the additional problem of having to first estimate $\Theta_0$ from data. The exposition in the remainder of this section and in Section (ref) focuses on binary outcomes. We will consider extensions to multinomial outcomes in Section (ref).

{\bf Known $\Theta_0$.} Given a decision space $\mathcal D \subseteq [0,1]$, a loss function $\ell : \{0,1\} \times \mathcal{D} \to \mathbb{R}_+$ and $\theta \in \Theta_0$, the risk of $ d \in \mathcal D$ under the forecast distribution $\mathbb{P}_\theta$ is \[ \mathbb E_\theta[ \ell(Y,d) ] = \ell(0,d) \, {\textstyle \mathbb{P}_\theta} ( Y = 0 ) + \ell(1, d) \, {\textstyle \mathbb{P}_\theta}( Y = 1 ) \,. \] The expectation $\mathbb E_\theta$ and forecast probabilities $\mathbb{P}_\theta$ may condition on covariates observed by the econometrician when making the forecast; we have suppressed this dependence to simplify notation. The $\theta$-optimal forecast, denoted $d_\theta^*$, minimizes risk under $\mathbb{P}_\theta$: \[ \mathbb E_\theta[ \ell(Y, d_\theta^* ) ] = \inf_{d \in \mathcal D} \mathbb E_\theta[ \ell(Y, d ) ] \,. \] A minimax forecast solves

The regret of a forecast is its risk in excess of the risk of the $\theta$-optimal forecast. A minimax regret forecast solves

Robust forecasts are derived under these criteria in Sections (ref) and (ref) for binary and multinomial outcomes, respectively.

{\bf Estimated $\Theta_0$.} In many scenarios the researcher might not know $\Theta_0$ and will therefore need to first estimate the set (or features of $\Theta_0$ germane to the forecasting problem) from a sample of data of size $n$.\footnote{The known-$\Theta_0$ case can be viewed as the limit as $n \to \infty$.} Our efficient robust forecasts deal with the additional uncertainty that arises from not knowing $\Theta_0$. Here “efficient robust” forecasts are those for which the maximum risk or regret is as close as possible to the maximum risk or regret of an oracle forecast with known $\Theta_0$ (see Section (ref)). This efficiency notion recognizes that uncertainty about the true identity of the forecast distribution among the class $\{\mathbb{P}_\theta : \theta \in \Theta_0\}$ is the dominant consideration asymptotically, but that estimation error may nevertheless have a material impact on the forecast in any finite sample.

To make the analysis of efficient robust forecasts tractable but reasonably general, we assume that the model for the data, say $X_n$, and outcome $Y$ is indexed by $\theta$ and a $k$-dimensional vector of reduced-form parameters $P \in \mathcal P \subseteq \mathbb R^k$. The parameters $\theta$ and $P$ are linked by a known mapping $P \mapsto \Theta_0(P)$. For partially identified forecasting models, $\Theta_0(P)$ denotes the identified set if $P$ was the true reduced form parameter value. We assume that $X_n$ and $Y$ are related to $\theta$ and $P$ in the following manner:

Condition ((ref)) implies that $Y$ does not depend on the data $X_n$ or $P$ beyond dependence through $\theta$. Condition ((ref)) says that the distribution of the data is fully summarized by $P$, which is standard for estimation and inference under partial identification; see, e.g., MoonSchorfheide2012. Examples 1-6 can be shown to fit this framework. Here we just discuss Example 1 for brevity.

\paragraph*{Example 1 (continued).} In this example, the econometrician observes data $X_n = (Y_i^T)_{i=1}^n$. The data are used to estimate the vector $P = (p(y^T))_{y^T \in \{0,1\}^T}$, which collects the probabilities of observing sequences $y^T \in \{0,1\}^T$. The mapping $\Theta_0(P)$ from $P$ to $\theta = (\beta,\Pi_{\lambda,y})$ is defined in ((ref)). Given a history $y^T$, the distribution of $Y \equiv Y_{iT+1}$ is fully summarized by $\theta$; see ((ref)). Moreover, the distribution of the data is itself multinomial over the different realizations of $y^T$ with probabilities $P$. Thus $F_{n,P}$ is the product of $n$ multinomial distributions with probabilities $P$. In this model $y_0$ and $\lambda$ are unit specific and the forecasts are constructed based on the posterior distribution of $(y_0,\lambda)$ conditional on $\theta = (\beta,\Pi_{\lambda,y})$, see ((ref)). $\Box$

Binary Forecasts

In this section we consider the binary forecasting problem. First, in Section (ref) we review several common binary loss functions and their corresponding $\theta$-optimal forecasts. In Section (ref) we derive forecasts that are robust to uncertainty about the forecast distribution and in Section (ref) we construct efficient robust forecasts for the case of an estimated $\Theta_0$. The forecasts are summarized in Table (ref) below. Section (ref) presents an application to semiparametric panel data models for dynamic binary choice.

$\theta$-Optimal Forecasts

We consider three loss functions to evaluate forecast accuracy: binary (or classification) loss, quadratic loss, and log predictive probability score.

\paragraph*{Binary (or Classification) Loss.} The binary loss function for $\mathcal D = \{0,1\}$ is

where $a_{10}, a_{01} \geq 0$. A special case with $a_{10} = a_{01}$ is classification loss $\ell_b(y,d) = \mathbb{I}[ y \neq d ]$.\footnote{When $a_{10} = a_{01}$, it is without loss of generality to normalize their common value to $1$.} The $\theta$-optimal forecast is

and its risk is

where $a \wedge b = \min\{a,b\}$. The $\theta$-optimal forecast is not unique when $\mathbb{P}_\theta(Y = 1) = \frac{a_{01}}{a_{01}+a_{10}}$. In this case, however, all $\theta$-optimal forecasts differ only in their handling of ties and have the same risk.

\paragraph*{Quadratic Loss.} The quadratic loss for $d \in \mathcal D = [0,1]$ is

With ${\cal D} = [0,1]$ the $\theta$-optimal forecast is the mean of $Y$ under the forecast distribution:

\paragraph*{\bf Log Loss.} Here the loss function for $\mathcal D = [0,1]$ is

The $\theta$-optimal forecast is also the mean:

Although the $\theta$-optimal forecasts under quadratic loss and log loss are the same, their risks are different: the risk under quadratic loss is the variance of the forecast distribution, whereas the risk under log loss is the entropy of the forecast distribution.

Robust Forecasts

We now relax the assumption that the forecast distribution $\mathbb{P}_\theta$ is known and derive forecasts that are robust with respect to $\mathbb{P}_\theta$ being any member of the set of forecast distributions $\{\mathbb{P}_\theta : \theta \in \Theta_0\}$. Note, however, that in this section we treat $\Theta_0$ as known.

The minimax and minimax regret forecasts will depend on the lower and upper values of the forecast probabilities as $\theta$ varies over $\Theta_0$:

Although our characterizations are general, the challenge in implementing robust forecasts is to solve these extremum problems which will typically require exploiting some additional structure. Appendix (ref) shows how duality methods may be used to simplify computation in a broad class of models that includes, but is not limited to, semiparametric dynamic binary choice models.

Minimax Forecasts

\paragraph*{Binary (or Classification) Loss.} We first derive the forecast that solves ((ref)) for the binary loss function $\ell_b$ from ((ref)) and decision space $\mathcal D = \{0,1\}$. The maximum risk of $d\in \{0,1\}$ is

The minimax forecast for binary (or classification) loss is therefore

and the minimax risk is \[ {\cal R}^*_{b,mm} =\left( a_{01}- a_{01}p_L \right) \, \wedge \, \left( a_{10} p_U \right) \,. \] The minimax binary forecast is not unique when $a_{01} = a_{01} p_L + a_{10} p_U$. In this case, each minimax forecast differs only in its handling of ties and has the same maximum risk.

\paragraph*{Quadratic Loss.} We now derive the forecast that solves ((ref)) for the quadratic loss function $\ell_q$ from ((ref)) and decision space $\mathcal D = [0,1]$. The maximum risk of $d \in [0,1]$ is

The minimax forecast is therefore

and the minimax risk is \[ {\cal R}^*_{q,mm} = \left[

\right. \]

\paragraph*{Log Loss.} The minimax forecast $d_{q,mm}$ is also minimax for the log loss function $\ell_p$ from ((ref)) and decision space $\mathcal D = [0,1]$; see Appendix (ref).

Minimax Regret Forecasts

\paragraph*{Binary (or Classification) Loss.} We first derive the forecast that solves ((ref)) for the binary loss function from ((ref)) and decision space ${\cal D} = \{0,1\}$. In view of ((ref)), the inner maximization problem in ((ref)) becomes

where $a_+ = \max\{a,0\}$. Therefore, the minimax regret forecast is

and its maximum regret is

The minimax and minimax regret binary forecasts for classification loss (i.e., $a_{01} = a_{10}$) are the same; see Appendix (ref). As with other forecasts for $\mathcal D = \{0,1\}$, the minimax regret forecast is not necessarily unique. Non-uniqueness arises whenever $( \frac{a_{01}}{a_{01} + a_{10}} - p_L )_+ = ( p_U - { \frac{a_{01}}{a_{01} + a_{10}}} )_+$. If so, each minimax regret forecast has the same maximum regret and differs only in its handling of ties.

\paragraph*{Quadratic Loss.} We now derive the forecast that solves ((ref)) for the quadratic loss function $\ell_q$ from ((ref)) and decision space ${\cal D} = [0,1]$. By convexity, the maximum regret of $d \in [0,1]$ is

The minimax regret forecast for quadratic loss is therefore the midpoint of the extreme forecast probabilities: \[ d_{q,mmr} = \frac{p_L + p_U}{2} \] and the minimax regret is \[ {\cal R}^*_{q,mmr} =\left( \frac{p_U - p_L}{2} \right)^2\,. \nonumber \]

\paragraph*{Log Loss.} Finally, we derive the forecast that solves ((ref)) for the log loss function from ((ref)) and decision space ${\cal D} = [0,1]$. The minimax forecast is the $\theta$-optimal forecast if $p_L = p_U$. Suppose $p_L < p_U$. The regret of any $d \in [0,1]$ is the Kullback--Leibler (KL) divergence \[ {\textstyle \mathbb{P}_\theta} ( Y = 1 ) \log \left(\frac{{\textstyle \mathbb{P}_\theta} ( Y = 1 )}{d}\right) + {\textstyle \mathbb{P}_\theta} ( Y = 0 ) \log \left(\frac{{\textstyle \mathbb{P}_\theta} ( Y = 0 )}{1-d}\right)\,. \] By convexity, the maximum regret must be obtained at either $p_L$ or $p_U$:

When $p = p_L$, term in parentheses is increasing for $d \geq p_L$ and when $p = p_U$ the term in parentheses is decreasing for $d \leq p_L$. The maximum regret is therefore minimized by choosing $d$ to equate the two values. The minimax regret forecast under the log-scoring rule $d_{p,mmr}$ uniquely solves

where $h(p) = - p \log p - (1-p) \log (1-p)$ is the entropy of a Bernoulli distribution with success probability $p$. The minimax regret forecast is therefore the value $p$ that minimizes the maximum KL divergence between the Bernoulli distribution and the forecast distribution $\mathbb{P}_\theta$ over $\theta \in \Theta_0$.

Efficient Robust Forecasts

We now dispense with the assumption that $\Theta_0$ is known. We consider the setting described at the end of Section (ref) in which the econometrician wishes to forecast $Y$ having observed data $X_n$. In order to develop an optimality theory, we evaluate forecasts by their {\em integrated maximum risk}, defined as

We will use a similar definition of {\em integrated maximum regret}, denoted by ${\cal B}_{mmr}(d_n;\pi)$. Here, $\pi$ is a prior distribution for $P$ with support ${\cal P}$, $\Pi_n(P|X_n)$ is the posterior distribution of $P$ after having observed the data $X_n$, and $F_n(X_n)$ is the marginal distribution of the data $X_n$. As in Section (ref), conditional on $P \in \mathcal P$, we consider the maximum risk or regret over $\Theta_0(P)$. This is the maximum risk faced by the forecaster if $P$ were the true reduced-form parameter. We then average across the joint distribution of $(P,X_n)$ to obtain the integrated maximum risk. The factorization of the joint distribution in the conditional distribution $\Pi_n(P|X_n)$ and the marginal distribution $F_n(X_n)$ highlights the well-known result that the integrated (maximum) risk is minimized by choosing the forecast that minimizes the {\em posterior (maximum) risk} for each realization $X_n$. We denote the optimal forecasts under the risk and regret objectives by $d_{b,mm}$ and $d_{b,mmr}$, respectively. We refer to them as {\em Bayesian robust forecasts} and derive explicit formulas in the remainder of this subsection.

Minimax Forecasts

We now make dependence of $p_L$ and $p_U$ on the reduced-form parameter explicit by writing

\paragraph*{Binary (or Classification) Loss.} In view of ((ref)), the posterior average maximum risk of choosing $d \in \mathcal \{0,1\}$ is

The Bayesian robust forecast is therefore

\paragraph*{Quadratic Loss.} In view of ((ref)), the posterior average maximum risk of choosing $d \in [0,1]$ is \[ \int \sup_{\theta \in \Theta_0(P)} \; \mathbb{E}_\theta [ \ell_q(Y, d) ] \, \mathrm d \Pi_n(P|X_n) = \left[

\right. \] The Bayesian robust forecast is therefore

\paragraph*{Log Loss.} The forecast $d_{q,mm}(X_n)$ is also the Bayesian robust forecast for the log loss function $\ell_p$ from ((ref)) and decision space $\mathcal D = [0,1]$; see Appendix (ref).

Minimax Regret Forecasts

\paragraph*{Binary Loss.} In view of ((ref)), the posterior average maximum regret for $d \in \{0,1\}$ is

The Bayesian robust forecast is therefore

\paragraph*{Quadratic Loss.} In view of ((ref)), the posterior average maximum risk of choosing $d \in [0,1]$ is \[ \int (p_U(P) - d)^2 \mathbb I \left[ d < \frac{p_L(P) + p_U(P)}{2} \right] + (p_L(P) - d)^2 \mathbb I \left[ d \geq \frac{p_L(P) + p_U(P)}{2}\right] \, \mathrm d \Pi_n(P|X_n) \,. \] The Bayesian robust forecast $d_{q,mmr}(X_n)$ is the minimizing value for $d \in [0,1]$, which can be computed numerically (e.g. by replacing the integral with the average across a large number of draws from the posterior then minimizing with respect to $d$).

\paragraph*{Log Loss.} In view of ((ref)), the posterior average maximum risk of choosing $d \in [0,1]$ is \[ \int \max_{p \in \{p_L(P), p_U(P)\}} \left(p \log \left( \frac{p}{d} \right) + (1-p) \log \left( \frac{1-p}{1-d} \right) \right) \, \mathrm d \Pi_n(P |X_n) \] The Bayesian robust forecast $d_{p,mmr}(X_n)$ is the minimizing value for $d \in [0,1]$, which again can be computed numerically.

Summary

Table (ref) summarizes the $\theta$-optimal, the robust, and the Bayesian robust decisions under binary, quadratic, and log loss functions.

Numerical Illustration

We close this section with a numerical illustration to show how uncertainty about the true $\theta \in \Theta_0$ can induce substantial variation in the implied forecast distribution in nonlinear models. We use a panel probit design from HonoreTamer2006. The model is as in Example 1 with $\Phi_t$ taken to be the standard normal cdf for all $t$. The distribution $\Pi_{\lambda,y}$ is unspecified, but $\lambda$ is assumed to be supported on the discrete evenly-spaced grid $\{-3,-2.8,\ldots,2.8,3\}$. Under the true data-generating process, $\lambda$ and $Y_{i0}$ are independent with $Y_{i0}$ taking the value $0$ or $1$ with probability $\frac{1}{2}$ and the probability mass for $\lambda$ is assigned by interpolating a $N(0,1)$ distribution on the support points.\footnote{See p.619 in HonoreTamer2006 for details.}

In this example, we wish to forecast $Y_{iT+1}$ having observed $Y_i^T$. In our earlier notation, the outcome of interest $Y$ represents $Y_{iT+1}$ and the forecast probability $\mathbb{P}_\theta(Y = 1)$ denotes the probability under $\theta$ that $Y_{iT+1} = 1$ given $Y_i^T$; see display ((ref)). The identified set $\Theta_0$ consists of all $(\beta,\Pi_{\lambda,y})$ that can match the model-implied probabilities of observing sequences $Y_i^T = y^T$ with the true probabilities for all $y^T \in \{0,1\}^T$; see display ((ref)).

We may compute the set of forecast probabilities $\{\mathbb{P}_\theta(Y = 1) : \theta \in \Theta_0\}$ by adapting linear programming methods from HonoreTamer2006 as follows. Denote the support of $\Pi_{\lambda,y}$ by $(\lambda_1,y_{01}), \ldots, (\lambda_L,y_{0L})$. As the support of $\Pi_{\lambda,y}$ is discrete, we identify $\Pi_{\lambda,y}$ with a $L$-vector $\pi \in \Delta^{L-1} := \big\{ x \in \mathbb R^L_+ : \sum_{l=1}^L x_l = 1 \big\}$ and write the restrictions defining $\Theta_0$ in display ((ref)) as $G(\beta) \pi = r$. The matrix $G(\beta)$ is a $2^T \times L$ matrix whose $l^{\mathrm{th}}$ column $G_l(\beta)$ is the $2^T$-vector of model-implied probabilities of observing different realizations of $Y_i^T$ conditional on $\lambda_i = \lambda_l$ and $Y_{i0} = y_{0l}$: \[ G_l(\beta) = \Big( p(y^T | y_{0l},\lambda_l; \beta ) \Big)_{y^T \in \{0,1\}^T} \,, \] where

and $r = \big(p(y^T)\big)_{y^T \in \{0,1\}^T}$ is the $2^T$-vector that collects the true probabilities of each realization of the sequences $Y_{i}^T$. The forecast probability ${\textstyle \mathbb{P}_\theta} ( Y_{iT+1} = 1 |Y_i^T =y^T )$ from display ((ref)) may also be written as $b(\beta)'\pi$ where $b(\beta)$ is an $L$-vector whose $l$\textsuperscript{th} entry is \[ b_l(\beta) = \frac{ \Phi(\beta y_T + \lambda_l) p(y^T | y_{0l},\lambda_l; \beta ) }{ p(y^T ) } \,, \] where $y_T$ denotes the element of $y^T$ corresponding to date $T$. Note that we can place $p(y^T)$ in the denominator because $\int p(y^T | y_0,\lambda; \beta ) \mathrm{d}\Pi_{\lambda,y}(\lambda,y_0) =p(y^T)$ for any $\theta \in \Theta_0$.

The problem of computing $p_U$ can be written as \[ p_U = \sup_\beta \left( \sup_{\pi \in \Delta^{L-1}} b(\beta)' \pi \quad \mbox{s.t.} \quad G(\beta) \pi = r \right) \,, \] with the understanding that the value of the inner maximization over $\pi$ is $-\infty$ if there does not exist a $\pi$ for which $G(\beta) \pi = r$. For such values of $\beta$ there does not exist a $\Pi_{\lambda,y}$ such that the model can explain the observed probabilities up to date $T$. As shown in Appendix (ref), the inner optimization over $\Pi_{\lambda,y}$ can be rewritten as a linear program, leading to the equivalent representation

where $K = 2^T$ and $A(\beta) = \left[ G(\beta)' - (1_{L \times 1} \otimes r') \, , \, - 1_{L \times 1} \right]$ with $\otimes$ denoting the Kronecker product. The lower value is computed similarly; see Appendix (ref).

Suppose $T = 2$ and the true $\beta_0 = 0.2$. The identified set for $\beta$ is approximately $[-2.4403,1.2428]$; these are all values of $\beta$ for which there is a $\Pi_{\lambda,y}$ such that the model can explain the observed probabilities up to date $T = 2$. The limits of the identified set for $\beta$ are denoted as grey dashed vertical lines in Figure (ref). For each value of $\beta$ in this set, we compute the smallest and largest values of the forecast probability $\mathbb{P}_\theta(Y = 1)$ subject to the constraint that $(\beta,\Pi_{\lambda,y}) \in \Theta_0$. The linear programming problem to compute the upper probability conditional on $\beta$ is characterized in parentheses on the right-hand side of ((ref)). The range of forecast probabilities as a function of $\beta$ is shown as the shaded regions in Figure (ref) for different values of $Y_{iT}$. Maximizing and minimizing with respect to $\beta$ yields the values $p_L$ and $p_U$; these are marked as black dotted horizontal lines in Figure (ref). Although each $(\beta, \Pi_{\lambda,y}) \in \Theta_0$ induces identical distributions over $Y_i^T$, they induce different distributions over $Y_{iT+1}$ and therefore different $\theta$-optimal forecasts. As a consequence, some parameterizations that are indistinguishable based on $T$ observations become distinguishable based on $T+1$ observations and the identified set shrinks over time. This feature is due to the sequential learning about heterogeneous parameters that generates a non-Markovian structure of the model.

In this numerical example, we consider robust forecasts which take $\Theta_0$ as known. This is the asymptotic problem faced by the forecaster in a large-$n$, fixed-$T$ setting. Suppose we condition on $Y_i^T$ with $Y_{iT} = 0$.\footnote{In this design, the conditional distribution of $Y_{iT+1}$ given $Y_i^T$ depends only on $Y_{iT}$. } The set of forecast probabilities $\{\mathbb{P}_\theta(Y = 1) : \theta \in \Theta_0\}$ is wide, spanning from $p_L = 0.2997$ to $p_U = 0.6803$ (see the left panel of Figure (ref)). In particular, there are $\theta \in \Theta_0$ for which $\mathbb{P}_\theta(Y = 1) < \frac{1}{2}$ so the $\theta$-optimal decision would be $d_{b,\theta}^* = 0$ for these $\theta$. However, there are other $\theta \in \Theta_0$ for which $\mathbb{P}_\theta(Y = 1) > \frac{1}{2}$ and therefore the corresponding $\theta$-optimal decision would be $d_{b,\theta}^* = 1$ for these $\theta$. Our robust forecasts are useful here as the forecaster has no way to discriminate among $\theta \in \Theta_0$ based on date-$T$ information. As $p_L + p_U < 1$, the minimax and minimax regret forecast for symmetric binary loss is therefore $d_{b,mm} = d_{b,mmr} = 0$. Similarly, when $Y_{iT} = 1$ the set of forecast probabilities $\{\mathbb{P}_\theta(Y = 1) : \theta \in \Theta_0\}$ is again quite wide, spanning $p_L = 0.3775$ to $p_U = 0.7320$ (see the right panel of Figure (ref)). Here $p_L + p_U > 1$ so $d_{b,mm} = d_{b,mmr} = 1$.

Multinomial Forecasts

We now extend the preceding analysis to multinomial forecasts. We first describe $\theta$-optimal forecasts with known $\theta$ (Subsection (ref)), then describe forecasts that are robust with respect to the set of forecasting models $\{\mathbb{P}_\theta : \theta \in \Theta_0\}$ with $\Theta_0$ known (Subsection (ref)), before concluding with efficient robust forecasts that deal with both model and sampling uncertainty (Subsection (ref)). The forecasts are summarized in Table (ref) at the end of this section.

$\theta$-optimal Forecasts

Throughout this section we focus on classification loss for the decision space $\mathcal{D} = \{0,1,\ldots,M\}$:

This loss function generalizes binary loss in the symmetric case (i.e., $a_{01} = a_{10}$) to multinomial forecasts.\footnote{It is straightforward to modify what follows to penalize some types misclassifications more heavily than others, as we did in the binary case. We adopt the equal-weighted specification ((ref)) for notational convenience.} The $\theta$-optimal forecast in this environment under $\mathbb{P}_\theta$ is the most likely outcome:

In the above display we write “$\in$” to allow for the possibility of ties. When the $\arg \max$ is non-singleton, any element of the set of minimizers is a $\theta$-optimal point forecast. Any $\theta$-optimal forecast has risk \[ 1 - \max_m \, {\textstyle \mathbb{P}_\theta} ( Y = m )\,. \]

Robust Forecasts

We now derive the minimax and minimax regret forecasts that solve the decision problems ((ref)) and ((ref)) for the classification loss function $\ell_c$ from ((ref)) and decision space $\mathcal D = \{0,1,\ldots,M\}$.

Minimax Forecasts

In the multivariate case, the analogues of $p_L$ and $p_U$ are the $M+1$ quantities

Computation of $\underline p_m$ using duality methods is discussed in Appendix (ref). The maximum risk from choosing $d \in \mathcal D$ is

The minimax forecast for classification loss is therefore

and the minimax risk is

As before, the minimax-optimal forecast is not necessarily unique. Non-uniqueness arises when the set of maximizers of $m \mapsto \underline p_m$ is not a singleton. If so, each minimax-optimal forecast differs only in its handling of ties and has the same maximum risk.

Minimax Regret Forecasts

For minimax regret forecasts, define

Suppose the forecaster chooses $d$. The difference $\max_{m'} \mathbb{P}_\theta ( Y = m' ) - \mathbb{P}_\theta ( Y = d ) $ is the regret from this choice under the forecast distribution $\mathbb{P}_\theta$. Having chosen $d$, the quantity $\Delta p_d$ is therefore the forecaster's maximum regret over all $\theta \in \Theta_0$. The minimax regret forecast is therefore \[ d_{c,mmr} \in \arg \min_{m} \Delta p_m \,. \] and the minimax regret is \[ \mathcal R^*_{c,mmr} = \min_{m} \Delta p_m \,. \] Unlike the binary case, equivalence of minimax and minimax regret forecasts for classification loss no longer holds when $M \geq 2$; see Appendix (ref). Computation of $\Delta p_m$ is discussed in Appendix (ref).

Efficient Robust Forecasts

In this section we now drop the assumption that $\Theta_0$ is known and consider also the need to estimate features of $\Theta_0$ that are relevant for the forecasting problem from data. We consider the same setup and notation as developed for the binary case in Section (ref).

Minimax Forecasts

Here we make dependence on the reduced-form parameter explicit by defining \[ \underline p_{m}(P) := \inf_{\theta \in \Theta_0(P)} \mathbb P_\theta(Y = m)\,. \] In view of ((ref)), the posterior average maximum risk of choosing $d \in \mathcal D$ is \[ \int \sup_{\theta \in \Theta_0(P)} \mathbb{E}_\theta [ \ell_c(Y, d) ] \, \mathrm d \Pi_n(P|X_n) = 1 - \int \underline p_{d}(P) \, \mathrm d \Pi_n(P|X_n) \] The Bayesian robust forecast is therefore

In case of ties, any (possibly randomized) tie-breaking rule is optimal. For instance, one could simply choose the smallest value of $m$ among the set of maximizers.

Minimax Regret Forecasts

For minimax regret forecasts, define \[ \Delta p_m(P) := \sup_{\theta \in \Theta_0(P)} (\max_{m'} \mathbb P_\theta(Y = m') - \mathbb P_\theta(Y = m))\,. \] The posterior average maximum regret of choosing $d \in \mathcal D$ is \[ \int \Delta_{p_d}(P) \, \mathrm d \Pi_n(P|X_n) \,. \] The Bayesian robust forecast is therefore

In case of ties, any (possibly randomized) tie-breaking rule is optimal. For instance, one could simply choose the smallest value of $m$ among the set of minimizers.

Asymptotic Efficiency for the Robust Forecasting Problem

In this section we focus on forecasts that are asymptotically efficient-robust. We continue to evaluate the forecasts by their {\em integrated maximum risk} (or regret), but only require this criterion to be minimized in the limit as the sample size $n$ tends to infinity. This enlarges the class of efficient forecasts to those that are asymptotically equivalent to the Bayesian robust forecast.

Some interesting findings emerge. First, “plug-in” rules, in which an efficient estimator $\hat P$ is plugged into the rules derived in Sections (ref) and (ref), are not asymptotically efficient-robust if the key quantities which determine the robust forecast (i.e., $p_L(P)$ and $p_U(P)$ in the binary case) are only directionally differentiable functions of $P$. This stands in contrast with other asymptotic efficiency results for related problems that depend smoothly on first-stage estimators, including point estimation under partial identification Song2014 and efficient statistical treatment rules under point identification HiranoPorter2009, for which plug-in rules are efficient. Second, forecasts that are constructed via {\em bagging} tend to be asymptotically efficient-robust. To construct such forecasts, the posterior distribution for $P$ is replaced by the bootstrap distribution of an efficient estimator of $P$. The forecast is then chosen to minimize the maximum risk or regret over $\Theta_0(P)$ averaged across the bootstrap distribution. As discussed in Remark (ref) below, it can be shown that the bagged forecasts are asymptotically equivalent to the Bayesian robust forecasts even under directional differentiability.

Limit Experiment

Our approach follows HiranoPorter2009 and uses Le Cam's limits of experiments framework. As is standard for treatments of asymptotic efficiency (see, e.g., Vandervaart2000), we work with a local reparameterization in which the reduced form parameter is $P_{n,h} = P_0 + n^{-1/2}h$ for $P_0$ fixed and $h$ ranging over $\mathbb R^k$. Let $\overset{P_{n,h}}{\rightsquigarrow}$ and $\overset{P_{n,h}}{\to}$ denote convergence in distribution and in probability under the sequence of measures $\{F_{n,P_{n,h}}\}_{n \geq 1}$. The model for $X_n$ is locally asymptotically normal at $P_0$ if for each $h_0 \in \mathbb R^k$, the likelihood ratio processes indexed by any finite subset $H \subset \mathbb R^k$ converge weakly to the likelihood ratio in a shifted normal model:

with $Z \sim N(h_0,I_0^{-1})$ and $I_0$ nonsingular. Let $\mathbb E_h$ and $\mathbb P_h$ denote expectation and probability with respect to $Z \sim N(h,I_0^{-1})$.

In the dynamic binary choice example, Assumption (ref).1. implies we observe all possible realizations of histories $Y_i^T \in \{0,1\}^T$ up to time $T$ with positive probability.

To describe the limit experiment, consider the collection $\mathbb D$ of sequences of forecasts $\{d_n\}_{n \geq 1}$ that converge in distribution under $\{F_{n,P_{n,h}}\}_{n \geq 1}$:

where $Q_{P_0,h}$ denotes a probability measure on $\mathcal D$ equipped with its Borel $\sigma$-algebra. Assumption (ref) permits application of an asymptotic representation theorem of Vandervaart1991. For any $\{d_n\}_{n \geq 1} \in \mathbb D$ there exists a function $d^\infty_{P_0}(Z,U) $ with $d^\infty_{P_0}(Z,U) \sim Q_{P_0,h}$ where $Z \sim N(h,I_0^{-1})$ and $U \sim\;$Uniform$[0,1]$ independently of $Z$ is a randomization term. As $n \to \infty$, the average excess maximum risk and regret of any sequence of forecasts $\{d_n\}_{n \geq 1} \in \mathbb D$ converges to a limiting counterpart for its representation $d^\infty_{P_0}(Z,U)$ in the limit experiment.

Asymptotic Efficiency

The robust forecasts derived in Sections (ref) and (ref) are oracle forecasts in the sense that they were obtained under the assumption of knowledge of the true set $\Theta_0$. To distinguish the oracle forecasts from the data-dependent forecasts $d_{mm}(X_n)$ and $d_{mmr}(X_n)$, in what follows we use the notation $d_{mm}^o(P)$ and $d_{mmr}^o(P)$ to denote the oracle forecasts when $P$ is the true reduced-form parameter. To facilitate the asymptotic calculations, we evaluate forecasts by their excess maximum risk or regret relative to the oracle. The excess maximum risk of $d_n(X_n)$ is \[ \Delta\mathcal R_{mm}(d_n; P, X_n) = \sup_{\theta \in \Theta_0(P)} \mathbb E_\theta[ \ell(Y,d_n(X_n))] - \sup_{\theta \in \Theta_0(P)} \mathbb E_\theta[ \ell(Y,d_{mm}^o(P))] \,. \]

Integration over $(P,X_n)$ leads to the {\em integrated excess maximum risk} \[ \Delta {\cal B}_{mm}^n(d_n;\pi) = \int \int \sqrt{n} \Delta\mathcal R_{mm}(d_n; P, X_n) \, \mathrm d\Pi_n(P|X_n) \, \mathrm dF_n(X_n). \] We standardize by $\sqrt{n}$ and recenter at the maximum risk of the oracle to ensure that $\Delta {\cal B}_{mm}^n(d_n;\pi)$ converges to a finite but potentially non-zero limit, though this does not change the ranking of forecasts. Therefore, the Bayes robust forecast under minimax risk also minimizes $\Delta {\cal B}_{mm}^n(d_n;\pi)$. Excess maximum regret $\Delta\mathcal R_{mmr}$ and integrated excess maximum regret $\Delta {\cal B}_{mmr}^n(d_n;\pi)$ are defined similarly, replacing risk in the above display with regret.

To derive the asymptotic counterparts, we begin by calculating a frequentist risk that averages over the data $X_n$ conditional on $P$ and then integrates over $P$ using the prior $\pi$. To express the frequentist risk, let $\mathbb{E}_{P_{n,h}}$ denote the expectation with respect to $X_n \sim F_{n,P_{n,h}}$. Using this notation and conducting the change-of-variables from $P_{n,h}$ to $h$, the {\em frequentist excess maximum risk} can be expressed as

Here we dropped the Jacobian term that arises from the change-of-variables because it simply scales the average excess maximum risk by a power of $n$ without changing the ranking of forecasts. We include $P_0$ in the conditioning set to indicate that the calculations are done locally around $P_0$. The regret $\Delta \mathcal B_{mmr}^n(d_n;P_0,\pi)$ can be expressed in a similar manner. asymptotically efficient-robust forecasts are those that minimize $\lim_{n \to \infty} \Delta \mathcal B_{mm}^n(d_n;P_0,\pi)$ and $\lim_{n \to \infty} \Delta \mathcal B_{mmr}^n(d_n;P_0,\pi)$. Under conditions permitting the interchange of limits and integration, we obtain \[ \lim_{n \to \infty} \Delta \mathcal B_{mm}^n(d_n;P_0,\pi) = \pi(P_0) \int \left( \lim_{n \to \infty} \mathbb E_{P_{n,h}}\left[ \sqrt n \Delta\mathcal R_{mm}\left(d_n, P_{n,h}; X_n\right) \right] \right) \, \mathrm d h\,, \] and similarly for regret. The limit in parentheses will depend on the sequence of forecasts $\{d_n\}_{n \geq 1}$ through its representation in the limit experiment. Also note that the asymptotic ranking of forecasts does not depend on $\pi$.

The following example illustrates the normalization of the excess maximum risk and the use of the local reparameterization.

\paragraph*{Example 7: Local parameters and frequentist excess maximum risk.} \phantom Suppose that $\mathcal P = (0,1)$, $p_L(P) = P$, and \[ p_U(P) = \left[

\right. \] For a binary loss function with $a_{01} = a_{10} = 1$, the robust forecast takes the form

If the true $P$ is bounded away from $\frac{1}{2}$, then eventually we will learn whether it is less or greater than $\frac{1}{2}$ and make the optimal decision. The most challenging case is when $P$ is very close to $\frac{1}{2}$. Thus, we center the local reparameterization at $P_0=\frac{1}{2}$. Suppose that under $P_{n,h}$ the frequentist sampling distribution and the Bayesian posterior for $P$ are \[ \hat P|P_{n,h} \sim N(P_{n,h},n^{-1}) \,, \quad P|X_n \sim N(\hat{P},n^{-1}) \,, \] respectively. The posterior is obtained under a uniform prior on ${\cal P}$ when $n$ is large enough so that the truncation effect of the prior at the boundary of $(0,1)$ is negligible. Under the local reparameterization, the sampling distribution of $\hat{h} = \sqrt{n}(\hat P-P_0)$ and the posterior distribution for $h = \sqrt n (P - P_0)$ are \[ \hat{h}|(P_0,h_0) \sim N(h_0,1) \,, \quad h|(X_n,P_0) \sim N(\hat{h},1) \,, \] respectively.

In this example $\hat P$ (equivalently $\hat{h}$) is a sufficient statistic. For any decision $d_n(\hat{h})$, we obtain

Here $1 - p_L(P_{n,h_0})=1/2 - n^{-1/2} h_0$ is linear in $h_0$ whereas \[ p_U(P_{n,h_0}) = \left\{

\right. \] Using straightforward algebra it can be shown that

where $\mathbb P_{n,h_0}$ denotes probabilities under $F_{n,P_{n,h_0}}$. It follows that for any sequence $\{d_n\} \in \mathbb D$, \[ \lim_{n \to \infty} \mathbb{E}_{P_{n,h_0}}[\sqrt{n}\Delta {\cal R}_{mm} (d_n(\hat{h}),P_{n,h_0};\hat{h})] = \left\{

\right. \] where $Z \sim N(h_0,1)$ under $\mathbb P_{h_0}$.\footnote{Here it is without loss of generality to write $d^\infty_{P_0}$ as a function of $Z$ only; see the discussion in Appendix (ref).} The formula shows that the $\sqrt{n}$ standardization leads to a well-defined non-trivial limit of the frequentist excess maximum risk. $\Box$

Binary forecasts

In this section we show that the efficient robust binary forecasts that were derived in Section (ref) are optimal. For brevity, we focus on discrete forecasts with $\mathcal D = \{0,1\}$ under binary or classification loss. This class of forecasts is also relevant for its connections with statistical treatment rules.

Say $f : \mathcal P \to \mathbb R^d$ is directionally differentiable at $P_0$ if the limit \[ \lim_{t \downarrow 0} \frac{f(P_0 + t h) - f(P_0)}{t} =: \dot f_{P_0}[h] \] exists for every $h \in \mathbb R^k$, in which case $ \dot f_{P_0}[\cdot]$ is its directional derivative. Note that $ \dot f_{P_0}[\cdot]$ will be positively homogeneous of degree one but not necessarily linear. If $\dot f_{P_0}[h]$ is linear in $h$ we say that $f$ is fully differentiable at $P_0$. We say that the posterior $\Pi_n$ is consistent if $ \Pi_n(P \in N |X_n) \overset{P_0}{\to} 1$ for every neighborhood $N$ containing $P_0$ for each $P_0 \in \mathcal P$. Recall that $Z \sim N(h,I_0^{-1})$ in the limit experiment. Let $\mathbb P_h$ denote probability statements with respect to $Z$. Moreover, let $Z^* \sim N(0,I_0^{-1})$ independently of $Z$ and $\mathbb E^*$ denote expectation with respect to $Z^*$.

The directional differentiability of Assumption (ref).1 was built into the functional form of $p_U(\cdot)$ in Example 7. Appendix (ref) shows that in a broad class of problems the extreme probabilities $p_L(P)$ and $p_U(P)$ can be expressed as min-max or max-min problems, where the outer optimization is over homogeneous parameters and the inner optimization is a linear or convex program. It follows that $p_L(\cdot)$ and $p_U(\cdot)$ are typically only directionally, rather than fully, differentiable functions.\footnote{See, e.g., Theorem 3.1 of Greenberg1997 for directional differentiability of the value of linear programs, Chapter 4.3 of BonnansShapiro2000 for directional differentiability of the value of convex programs, and MilgromSegal2002 and Shapiro2008 for directional differentiability of min-max problems.} Directional differentiability can also be a feature of models defined via moment inequalities (cf. Example 5).

Assumption (ref).2(a) holds under standard regularity conditions (see, e.g., Chapter 10.4 of Vandervaart2000). For Assumption (ref).2(b), note that $\Pi_n(P \not \in N|X_n) $ typically converge to zero exponentially. For instance, in a normal means model with $\bar X_n \sim N(P_{n,h},(nI_0)^{-1})$ and a flat prior on $P$, we have $\Pi_n(P \not \in N|X_n) = O(e^{-c n})$ for some $c > 0$.\footnote{Note $P | X_n \sim N(\bar X_n,(nI_0)^{-1})$ under a flat prior. Choose $\varepsilon > 0$ so that $B_{2\varepsilon}(P_0) \subset N$. Then $P_0(\bar X_n \in B_\varepsilon(P_0)) \to 1$ and whenever $\bar X_n \in B_\varepsilon(P_0)$, we have $\Pi_n(P \not \in N|X_n) \leq \Pi_n(|P - \bar X_n| > \varepsilon|X_n) = O(e^{-c n})$ for any $c < \frac{\varepsilon^2}{2} \lambda_{\min}(I_0)$.} More generally, the classical posterior consistency results of Schwartz1965 establish exponential convergence rates.

Assumption (ref).3 is simply assuming that a $\delta$-method applies for the posterior distribution of directionally differentiable functionals of $P$.\footnote{See, e.g., KitagawaOleaPayneVelez2020 for a formal justification. This may require strengthening our definition of directional differentiability to Hadamard directional differentiability.} For a heuristic justification for Assumption (ref).3(a), consider a normal means model with $\bar X_n \sim N(P_{n,h},(nI_0)^{-1})$. Under a flat prior for $P$, we have $P | X_n \sim N(\bar X_n,(nI_0)^{-1})$. For a directionally differentiable function $f$:

with the final line is by the change of variables $\kappa = \sqrt n (P - \bar X_n)$. The last integral can be rewritten $\mathbb E^*[ \dot f_{P_0}[Z^* + Z] |Z ] $ where $Z^* |Z \sim N(0,I_0^{-1})$ with $Z = \sqrt n (\bar X_n - P_0) \sim N( h,I_0^{-1})$ under $F_{n,P_{n,h}}$. A similar argument provides a heuristic justification for Assumption (ref).3(b). In the presentation of the asymptotic efficiency results for the binary forecasts we rely on the following definition:

Let $\Delta \mathcal B_{b,mm}^n(\,\cdot\,;P_0,\pi)$ and $\Delta \mathcal B_{b,mmr}^n(\,\cdot\,;P_0,\pi)$ denote integrated excess maximum risk and regret (see display ((ref))) for binary loss $\ell_b$ from ((ref)). We require forecasts to satisfy an additional technical condition, namely condition ((ref)) in the Appendix, which permits the interchange of limits and integration. This condition can be verified under more primitive conditions (see Remark (ref)). Let $\mathbb D$ denote the set of all sequences of $\{0,1\}$-valued forecasts that converge in the sense of ((ref)). Theorem (ref) states that forecasts that are asymptotically equivalent to the Bayes forecasts are asymptotically optimal. A proof is provided in Appendix (ref).

As we formalize in Proposition (ref) below, asymptotic equivalence to $d_{b,mm}$ (respectively, $d_{b,mmr}$) is a necessary condition for a forecast $\tilde d_n$ to be asymptotically efficient-robust under minimax risk (respectively, regret) under a side condition ensuring that ties occur with probability zero.

In view of Definition (ref), we say that asymptotic equivalence fails at $P_0$ if there exists some $h_* \in \mathbb R^k$ for which $d_n(X_n)$ and $d_{b,mm}(X_n)$ have different asymptotic distributions under the sequence of measures $\{F_{n,P_{n,h_*}}\}_{n \geq 1}$ with $P_{n,h_*} = P_0 + n^{-1/2} h_*$. A proof for the following proposition is provided in Appendix (ref).

\paragraph*{Example 7: (continued).} The oracle forecast under the minimax risk criterion was given in display ((ref)). In order to compute the Bayesian robust forecast we need to evaluate

where $\hat h = \sqrt n (\hat P - P_0)$ with $P_0 = \frac{1}{2}$ and $\Phi_N$ and $\phi_N$ denote the standard normal cdf and pdf. As $\hat P$ is a sufficient statistic, the forecasts only depend on the data $X_n$ though $\hat P$ or, equivalently, through $\hat h$. We may deduce that

Note that the term on the right-hand side of the inequality in the indicator function is always negative. The plug-in forecast is obtained by replacing the unknown $P$ by $\hat P$, leading to $d^{plug}_{b,mm}(\hat P) = \mathbb I[p_L(\hat P) + p_U(\hat P) \geq 1]$. In the present example, the plug-in forecast can be expressed equivalently in terms of $\hat h$:

The plug-in forecast is not asymptotically equivalent to the Bayesian robust forecast. In particular, the Bayesian robust forecast predicts $Y = 1$ more aggressively than the plug-in forecast. It can be verified by direct calculation in this example that this more aggressive forecast is asymptotically efficient-robust. It also follows from Proposition (ref) that the plug-in forecast is not asymptotically efficient-robust. This is seen by noting that $a_{01} = a_{10} = 1$, $p_{L}(P_0) + p_U(P_0) = 1$, $\dot p_{L,P_0}[h] = h$, $\dot p_{U,P_0}[h] = 2(h)_+$, and so $\mathbb E^*\big[a_{01} \dot p_{L,P_0}[Z^* + Z] + a_{10} \dot p_{U,P_0}[Z^* + Z] \, \big| \,Z \big]$ reduces to $Z + 2\mathbb E^*[(Z^* + Z)_+ \,| \,Z ]$ which is nonzero almost everywhere.

To quantify the inefficiency of $d_{b,mm}^{plug}(\hat{h})$ relative to $d_{b,mm}(\hat{h})$, straightforward algebraic manipulations using ((ref)) allow us to derive formulas for the frequentist excess maximum risk of $d_{b,mm}(\hat{h})$ and $d_{b,mm}^{plug}(\hat{h})$ as a function of $h_0$. The results are plotted in Figure (ref). The plug-in forecast is inferior to the Bayesian robust forecast from an integrated risk perspective: the area under the curve corresponding to $d_{b,mm}$ is around 20% smaller than that under the curve corresponding to the plug-in forecast. While $d_{b,mm}$ was designed to be optimal from an integrated risk perspective, it also dominates the plug-in forecast from a minimax perspective: the maximum excess maximum risk of the plug-in forecast in the limit experiment is around 75% larger than that of $d_{b,mm}$.

Similar calculations can be made under the regret criterion. The oracle forecast is of the form $d^o_{b,mmr}(P) = \mathbb{I} [ (\frac{1}{2} - p_L(P) )_+ \leq ( p_U(P) - \frac{1}{2} )_+ ]$. Similar calculations as for the risk criterion can be used to obtain a formula for the Bayesian robust forecast. In turns out that the plug-in forecast remains unchanged. Frequentist excess maximum regrets as a function of $h_0$ are plotted in Figure (ref). Again, the plug-in forecast is not asymptotically efficient-robust and dominated by the Bayesian efficient robust forecast once we average across $h_0$. Its integrated excess maximum regret is around 8% smaller and maximum excess maximum regret is around 41% smaller. Also shown is the excess maximum regret of a forecast which plugs the posterior means of $p_L(P)$ and $p_U(P)$ into the oracle: $d^\dagger(X_n) = \mathbb{I} [ (\frac{1}{2} - \int p_L( P) \, \mathrm d \Pi_n(P|X_n) )_+ \leq ( \int p_U( P) \, \mathrm d \Pi_n(P|X_n) - \frac{1}{2} )_+ ]$. This forecast is equivalent to the minimax forecast $d_{b,mm}$ and is therefore optimal for minimizing integrated excess maximum risk but not necessarily integrated excess maximum regret. Figure (ref) shows that $d_{b,mmr}$ also dominates $d^\dagger$ in terms of both its average (2.5% smaller) and maximum (21% smaller) excess maximum regret in the limit experiment. $\Box$

Multinomial Forecasts

We now turn to extending the asymptotic efficiency result to multinomial forecasts that are asymptotically equivalent to the Bayesian robust forecast from Section (ref). To do so, we first state some additional regularity conditions.

Assumption (ref) is similar to Assumption (ref). In particular, a heuristic justification for Assumption (ref).3 follows similar reasoning to that presented earlier for Assumption (ref).3.

We now present the asymptotic efficiency results for multinomial forecasts. Let $\Delta \mathcal B_{c,mm}^n(\,\cdot\,;P_0,\pi)$ and $\Delta \mathcal B_{c,mmr}^n(\,\cdot\,;P_0,\pi)$ denote integrated excess maximum risk and regret (see display ((ref))) for classification loss $\ell_c$ from ((ref)). Also let $\mathbb D$ denote the set of all sequences of $\{0,\ldots,M\}$-valued forecasts that converge in the sense of ((ref)).

As with Remark (ref), bagged forecasts in which the posterior distribution is replaced with the bootstrap distribution of an efficient estimator of $P$ can be shown to be asymptotically efficient-robust under a suitable modification of the regularity conditions. As with Proposition (ref), it is possible to show that forecasts that are not asymptotically equivalent to the $d_{c,mm}$ and $d_{c,mmr}$ are not asymptotically efficient-robust under side conditions ruling out ties.

Conclusion

In this paper we proposed use of robust forecasts that are obtained by solving a minimax risk or minimax regret problem to deal with uncertainty about the forecast distribution. We also derived asymptotically efficient-robust forecasts that deal with the estimation of the set of forecast distributions. In addition to being useful for forecasting binary and multinomial outcomes, these methods have wide applicability in environments in which a forecaster is concerned about structural breaks, model misspecification, or a policy maker has to make treatment assignments.

{ \singlespacing }

\setcounter{page}{1}