On the robustness of posterior means

On the robustness of posterior means

Consider an observation $X$ whose likelihood is $X \mid \theta \sim \Norm(\theta, \sigma^2)$ for some known $\sigma^2$. There are two priors for $\theta$, denoted by $G_0$ and $G_1$. Suppose both priors have mean zero and have finite variances bounded by $V > 0$. Consider the decision problem of estimating $\theta$ under squared error, with $L(a,\theta) = (a-\theta)^2$. For the Bayesian with prior $G_1$, the Bayes decision rule is the posterior mean $\E_{G_1} [\theta \mid X]$ under the prior $G_1$. This decision rule attains Bayes risk under the prior $G_0$\[ R(G_1, \sigma; G_0) \equiv \E_{\theta \sim G_0}\bk{ (\E_{G_1}[\theta \mid X] - \theta)^2 }. \] We can think of $R(G_1, \sigma; G_0)$ as a measure of decision quality under disagreement. It measures the quality of $G_1$'s decision from $G_0$'s point of view. When $G_1 \neq G_0$, how large can $R (G_1, \sigma; G_0)$ be?

This note bounds $R(G_1, \sigma; G_0)$ uniformly over $G_1, G_0, \sigma$.\footnote {The previous version of this note (arXiv:2303.08653v1) claimed that $R (G_1, \sigma; G_0)$ is uniformly bounded over all $G_0, G_1, \sigma>0$, subjected to the constraints on the moments of $G_0,G_1$. Regrettably, it contained a critical error that rendered its proof incorrect. In particular, in that version, the display before (A1) on p.4 is incorrect: Posterior means of mixture priors are mixtures of posterior means under each mixing component, but the mixing weights are posterior probabilities assigned to each mixing component; thus, the mixing weights depend on the data rather than being fixed.

This version partially restores the main result in the previous version. (ref) shows that the maximum Bayes risk under $G_0$ is uniformly bounded over all $G_0, G_1, \sigma^2$ where $G_1$ satisfies an additional tail condition (ref). The bound we obtain depends on the tail condition, and thus (ref) is insufficient for the result in the previous draft. } Since \[ R(G_1, \sigma; G_0) \le 2\pr{ \E_{G_0}[\E_{G_1}[\theta \mid X]^2] + \E_{G_0}[\theta^2] } \le 2V + 2\E_{G_0}[\E_{G_1} [\theta \mid X]^2],\] it thus suffices to bound $\E_{G_0}[\E_{G_1}[\theta \mid X]^2]$ modulo constants. That is, it suffices to bound the 2-norm $\norm{\E_{G_1} [\theta \mid X]}^2$ under the law $X \sim \Norm(0,\sigma^2) \star G_0$.

The rest of the note shows that this quantity is uniformly bounded over all $G_0, G_1, \sigma^2 > 0$. Specifically, (ref) shows that for all $G_0, G_1$ that are mean zero and have variance bounded by $V$, $\E_{G_0}[\E_{G_1}[\theta \mid X]^2]$ is bounded by a constant that depends only on $(V, \sigma^2)$. This bound is large when $\sigma^2$ is large. To improve this bound, (ref) then shows that, if $G_1$ additionally satisfies some conditions on its tail behavior, $\E_ {G_0}[\E_ {G_1}[\theta \mid X]^2]$ is bounded by a constant that depends only on $V$ and the tail condition---and does not depend on $\sigma$.

To show a more powerful bound, we require a stronger condition on the tails of $G_1$ and derive bounds that are independent of $\sigma$ but are dependent on the tail conditions. In particular, assume \[ \max\pr{1-G_1(s), G_1(-s) }\le C_{G_1} s^{-k} \addtocounter{equation}{1}\tag{\theequation}\label{eq:g1_cond} \] for some $k > 2$ and $C_{G_1} > 0$, for all $s > 0$. We will also assume that $\E _{G_1}[\theta^2 \mid X]$ exists almost surely. Note that if $\E_{G_1} |\theta|^ {2+\epsilon} < m$, then $k$ can be taken to be $2+\epsilon$ and $C_{G_1}$ can be taken to be $m$ by Markov's inequality. In the rest of the proof, we let $C_t < \infty$ denote a positive constant that depends only on $t$. Occurrences of $C_t$ might correspond to different constant values.