Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
12,692 characters · 7 sections · 7 citation commands
The Bias-Variance Tradeoff in Long-Term Experimentation
As we exhaust methods that reduces variance without introducing bias ting2023limits, reducing variance in experiments often requires accepting some bias, using methods like winsorization wang2019measuring or surrogate metrics duan2021online. While this bias-variance tradeoff can be optimized for individual experiments, bias may accumulate over time, raising concerns for long-term optimization. We analyze whether bias is ever acceptable when it can accumulate, and show that a bias-variance tradeoff persists in long-term settings. Improving signal-to-noise remains beneficial, even if it introduces bias. This implies we should shift from thinking there is a single “correct”, unbiased metric to thinking about how to make the best estimates and decisions when better precision can be achieved at the expense of bias.
Furthermore, our model adds nuance to previous findings azevedo2020abtesting,sudijono2024optimizing that suggest less stringent launch criterion leads to improved gains. We show while this is beneficial when the system is far from the optimum, more stringent launch criterion is preferable as the system matures.
We study this problem by treating our long term optimization problem like a stochastic gradient descent (SGD) procedure welling2011bayesian. Each experiment can be treated as a noisy measurement of the lift, so each launch is a noisy step towards or away from the global optimum. This sets up a stochastic differential equation (SDE) that we can solve. By injecting bias and variance into the procedure, we can study whether the optimality gap increases or decreases.
For simplicity, we consider a problem where we optimize a metric by changing a parameter. When the parameter is $x_t$ at time $t$, the observed metric is $\mathrm{Metric}(x_t)$. This contains some sampling noise and has expectation $\operatorname{\mathbb{E}} \mathrm{Metric}(x_t)$, an objective that we wish to optimize. Without loss of generality, we can assume $\operatorname{\mathbb{E}} \mathrm{Metric}(x)$ attains the optimum at $x = 0$. Since at the optimum, the true objective has zero gradient, we can express the objective as \[ \operatorname{\mathbb{E}} \mathrm{Metric}(x_t) = \mathrm{optimum} - \frac{1}{2}r x_t^2, \] where $r$ captures the scaling of the effect of the parameter $x$ on the outcome.
At time $t$, with probability $\sfrac{1}{2}$ we propose to change the parameter $x_t$ to $x_t - 1$ and with probability $\sfrac{1}{2}$ we propose changing it to $x_t + 1$. A change from $x_t$ to $x_t - 1$ will change the objective by
which we can measure via an experiment. For simplicity, we assume all experiments to have standard error $\sigma$. The experiment is then launched if the result is statistically positive, so the probability of changing $x_t$ to $x_t - 1$ is \[ \operatorname{\mathbb{P}}(Z + rx_t / \sigma > c) = \bar\Phi(c - rx_t / \sigma), \] where $Z$ is a standard normal variable, $\bar\Phi$ is its survival function (or one minus its cumulative distribution function), and $c$ is the hypothesis testing threshold for a $z$-score at level $\alpha$, e.g.\ $c = 1.64$ for $90\%$ confidence. Similar, a change from $x_t$ to $x_t + 1$ is launched with probability $\bar\Phi(c + rx_t / \sigma)$. In other words, given the parameter is $x_t$ at time $t$, the parameter at time $t+1$ will be given by \[ X_{t+1} =
\]
Consider the case where it is difficult to detect a treatment effect, and the scale of noise is bigger than the effect size, i.e.\ $\sigma \gg |rx_t|$. The parameter $X_{t+1}$ will satisfy
After rescaling time to obtain a continuous time process, these define the drift and diffusion terms of an SDE:
where $\theta = r\phi(c) / \sigma$.
This is a well-known SDE for the Ornstein--Uhlenbeck process oksendal2003stochastic, i.e.\ a mean reverting walk. This is a particularly nice SDE cause it has a closed form solution for its distribution at every time point. The solution and limit distribution are given by
Correspondingly, the objectives at time $t$ and as $t \to \infty$ are
This equation already describes interesting properties of the stochastic optimization of systems through experimentation and highlight the importance of reducing the variance of lift estimates.
While the first point is completely unsurprising, the second and fourth suggest that as the system becomes more and more optimized, experiments need to be run with higher power and more stringent launch criterion to reach a better final state!
Perhaps counterintuitively, the third and fourth points fall in line with previous suggestions to relax the $p$-value threshold in azevedo2020abtesting,sudijono2024optimizing --- in a less mature system farther from the stationary state, effect sizes are bigger and a relaxed $p$-value threshold helps to achieve faster improvements; while in a mature system closer to the stationary state, effect sizes are smaller and a more stringent $p$-value threshold is preferred.
But what happens when you induce bias? For simplicity, we consider two scenarios:
In the first scenario, we should still converge to a limiting distribution with the right mean, while in the second scenario we assume the bias actually changes the mean of the limiting distribution.
In the first scenario, the true effect in (ref) becomes $rx_t / \gamma_r$ and the standard error becomes $\sigma / \gamma_\sigma$. The SDE ((ref)) can be updated by replacing $r$ with $r / \gamma_r$ and $\theta$ with $\frac{\gamma_\sigma}{\gamma_r} \theta$. By plugging this into the solution ((ref)), we can see that this scenario affects the solutions in how fast it converges, faster convergence is achieved when $\gamma_r < \gamma_\sigma$. In other words, it improves all aspects of the optimization if the signal-to-noise ratio improves so that $\frac{r / \gamma_r}{\sigma / \gamma_\sigma} > \frac{r}{\sigma}$. We will denote the improvement in signal-to-noise ratio as $\gamma$, i.e.\ $\gamma = \gamma_\sigma / \gamma_r$.
Furthermore, plugging this in (ref), the new expected target metric is \[ \operatorname{\mathbb{E}} \mathrm{Metric}(X_\infty) = \mathrm{optimum} - \frac{1}{4} \frac{r \alpha}{\gamma \theta}. \]
The key takeaway is that any improvement in the signal-to-noise ratio is beneficial, even at the expense of biased experiment results when the bias just shrinks the effect size.
Now consider a metric that is a biased measurement of the true estimand, e.g.\ when the estimand cannot be directly measured or if one uses a surrogate metric. This alternative metric induces another objective with the corresponding Ornstein--Uhlenbeck dynamics \[ \operatorname{\mathbb{E}} \mathrm{Metric}'(x_t) = \mathrm{optimum} - \frac{1}{2} r'(x - \mu)^2, \] where the optimizer of the objective is not zero but some value $\mu$. Suppose we can measure the changes in this alternative metric with standard error $\sigma'$. Using the same notation in (ref) where $\gamma$ stands for the improvement in the signal-to-noise ratio $\frac{r'}{\sigma'} / \frac{r}{\sigma}$, the solution becomes \[ X_t \sim N\left(x_0 e^{-\gamma \theta t} + \mu(1 - e^{-\gamma\theta t}), \frac{\alpha}{2\gamma\theta} \left(1 - e^{-2\gamma\theta t}\right)\right) \] and the true objective is \[ \operatorname{\mathbb{E}} \mathrm{Metric}(X_\infty) = \mathrm{optimum} - \frac{1}{2} r \left(\mu^2 + \frac{\alpha}{2 \gamma\theta}\right). \]
Here, the final state improves if $\mu^2 < \frac{\alpha}{2\theta} (1 - \gamma^{-1})$. This shows there is a bias-variance tradeoff in this scenario as well and that benefits rely only on improvements to the signal-to-noise ratio.
Furthermore, we note the relative importance of bias. While increasing bias has a quadratic impact on the final objective, increase the noise-to-signal ratio only has a linear effect. This suggests as systems become increasingly optimized, that surrogate metrics may need to be designed more carefully and favor reducing bias over reducing variance. How to empirically choose this tradeoff is still an open question.
Methods such as winsorization introduce bias for the sample mean, akin to the attenuation bias in the scenario in (ref) as the same bias is introduced to both the test and control groups. Our demonstration provides some justification for these long-term bias-variance tradeoffs.
We demonstrate there are bias-variance tradeoffs in long-term experimentation. While bias in the objective can be considered relatively more important than variance since the expected value of the objective at $X_\infty$ is quadratic in the bias $\mu$ and linear in the standard error $\sigma$, the notion that “unbiasedness means correctness” is incomplete.
\printbibliography