Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
88,644 characters · 20 sections · 89 citation commands
Optimizing Returns from Experimentation Programs
Experimentation in the private sector -- particularly in the online tech industry -- broadly differs from experimentation in scientific fields in two ways. Firstly, the goal of the vast majority of experiments in the private sector is to optimize a metric. For example, tech firms optimize measurable business objectives such as long-term retention, sign-up rates, or revenue. Secondly, repositories of past experiments give firms strong priors on treatment effects from related experiments. Empirical evidence azevedo2020b, goldberg2017decision, guo2020empirical, georgiev_1001, georgiev2022analysis, kohavi2024false suggests the following stylized fact regarding experiments: treatment effect distributions are typically centered around zero, with significant dispersion around the mean. Figure (ref) shows treatment effect densities for three distinct experimentation initiatives at Netflix, corroborating this point.
Given this prior on treatment effects, how should an organization conduct experimentation when the goal is to optimize a metric of interest? The current statistical paradigm in industry, null hypothesis statistical testing, can be mismatched with this goal: it is indifferent to utilities and opportunity costs, and it provides no principled justification for Type I and Type II error defaults manski2019treatment. Building on a growing literature azevedo2020b,azevedo2023b, guo2020empirical, goldberg2017decision, deng2015objective, we study an alternative Bayesian decision-theoretic framework and discuss how organizations should restructure current practice towards optimizing returns from experimentation.
An organization conducting experimentation typically has a pool of untested ideas and limited resources with which to conduct experiments. These organizations have a population of units on which to experiment -- an allocation pool -- and teams to generate new ideas for experiments and subsequently analyze them. Generally, experimentation in these firms proceeds as follows. A product team generates an idea which may improve a product. The idea is tested using a randomized controlled experiment or A/B test kohavi2009online. Based on the results of the A/B test, the idea is either discarded or shipped -- made available to all units in the allocation pool. Often, similar tests on the same product, platform, or population can be grouped into initiatives that we will call experimentation programs.
The A/B Testing Problem azevedo2020b provides a starting point for optimizing the returns from a single experimentation program. It is defined as follows. Suppose the experimentation program consists of $I$ untested ideas or interventions indexed by $i=1,\dots,I$. The data is not assumed to arrive sequentially. Treatment effects for these ideas are measured in the same metric. We will think of these effects $\Delta_i$ as the return associated with idea $i$. Data from past experiments give credence to a Bayesian assumption that $\Delta_i$ are drawn iid for some distribution $G$. We choose an allocation of units to A/B tests, $(n_i)_{i=1}^I$, such that $\sum_i n_i = N$, the total size of the allocation pool. A unit may only be assigned to one test at a time, making the allocation pool a limited resource.
For those tests with $n_i > 0$, we observe the result of the experiment, given by \[ \hat{\Delta}_i \sim \textsf{N}\left(\Delta_i, \frac{\sigma^2}{n_i} \right). \] Normality of the observations is justified by the central limit theorem; $\sigma$ represents typical fluctuations of the return metric, which is fairly stable and known in advance. The goal of the A/B testing problem is to select an allocation $(n_i)_{i=1}^I$ and a subset of tests $S$ to ship, $S \subseteq [I]$, in order to maximize total expected returns to experimentation,
where $u$ is an increasing utility function. We will typically take $u(x) = x$ to be risk-neutral. However $u$ may also capture risk-averse utilities such as $u(x) = x + bx\mathbf{1}\left\lbrace x < 0 \right\rbrace, b > 0$, inspired by prospect theory kahneman2013prospect.
The idea of optimizing returns from a portfolio of A/B tests is a stark alternative to traditional statistical analysis of experiments. The decision theoretic foundations of this perspective were first discussed in azevedo2020b, azevedo2023b, goldberg2017decision, manski2019treatment, guo2020empirical. These works also adopt a Bayesian or empirical Bayesian approach to the problem. See Appendix (ref) for an extended discussion of closely related work in the analysis of collections of A/B tests.
Most relevant to our work is the A/B Testing Problem in azevedo2020b, which is the first to recast experimentation as a constrained optimization problem -- exactly the A/B Testing Problem in the previous section. By a careful asymptotic analysis, azevedo2020b characterizes the optimal experimentation strategy in specific settings. For $t$-distributed priors which are sufficiently heavy-tailed, the optimal allocation is to split $N$ units equally across the $I$ tests, a strategy called lean experimentation: firms should run many low-powered tests. For light-tailed $t$-distributed priors $G$ with $N$ small enough, the optimal strategy is to go big: it is optimal to allocate all $N$ units to one test.
Collectively, the aforementioned literature azevedo2020b, azevedo2023b, goldberg2017decision, manski2019treatment, guo2020empirical and this paper represent a different approach to experimentation that we will refer to as the return-aware framework, to contrast it with null hypothesis testing. The framework views experimentation as a decision theoretic problem where the goal is to maximize the cumulative impact of experimentation, subject to various constraints.
This article makes original contributions to the return-aware framework and discusses its implications for current practice. We also argue that the return-aware framework is well-suited to experimentation in industry, and worthy of future research. First, we highlight several simple, useful observations about the framework.
\paragraph*{Practical Solution Methods.} The elegant results of azevedo2020b,azevedo2023b leave open the optimal allocation for general priors and arbitrary values of $I,N$. We show that dynamic programming solves this problem for any choice of $I,N$, utility function $u$, and prior $G$.
\paragraph*{Connection to $p$-value Frameworks.} In current experimentation practice, A/B tests are usually decided based on $p$-values and not posterior expectation calculations. We show that tuning the hypothesis testing level $\alpha$ based on the prior $G$ can replicate the optimal solution of Eq. (ref). As a result, there is little engineering work required to adopt the return-maximizing framework, and the implied optimal $p$-value gives insight into how current practice should be changed.
\paragraph*{Analytic Approximations to the Optimal Solution.} We give a simple approximation to optimal returns from an experimentation program using the optimal allocation and decision rule. The approximation shows how program-level returns depends on the size of the allocation pool, the number of ideas available for testing, and properties of the prior distribution.
\paragraph*{Implications for Current Experimentation Practice.} Second, we apply the previous three insights to experimentation programs at Netflix, a large online technology firm. Under the assumption of no costs, they suggest that current practice, justified by null hypothesis statistical testing, is highly suboptimal for the goal of optimizing returns. The optimal allocation of the A/B testing problem suggests many more tests should be run in practice. Moreover, the implied optimal one-sided $p$-values are found to be markedly larger than $0.05$. Incorporating costs gives a more nuanced picture that we discuss in Section (ref). Given the similarities in experimentation practice across technology firms, these results are likely applicable to other firms as well.
\paragraph*{Managing Multiple Experimentation Programs and Further Variants.} Thirdly, we discuss various extensions to the base A/B testing problem framework to managing multiple experimentation programs with shared resources. The extensions can be solved with dynamic programming and provide useful tools for managers to make investment decisions across several initiatives. \\
Finally, we end with important caveats on the return-aware framework and several directions for future research.
The basic solution to the A/B Testing Problem is given in azevedo2020b, which discusses both the optimal decision rule and resulting optimization problem describing the optimal allocation. We will focus exclusively on the case where $\mathbb{E}_G[\Delta] \leq 0$, which is a realistic assumption across many mature experimentation programs where most low hanging fruit has been picked azevedo2020b.
\sloppy Given the optimal decision, the allocation problem then reduces to maximizing \[ \sum_{i=1}^I \mathbb{E}\left[\mathbb{E}\left[ u(\Delta_i) | \hat{\Delta}_i;n_i \right]^+ \right] \] subject to a simplex constraint. The summand \[ f(n) := \mathbb{E}\left[\mathbb{E}\left[ u(\Delta) | \hat{\Delta};n \right]^+ \right] \] as a function of $n$ is called the production function azevedo2020b, which represents the expected return from testing a typical idea with $n$ units, using the optimal decision. When $n = 0$, we set $f(n) = 0.$ The production function can be computed with knowledge of the treatment effect distribution $G$ and $\sigma$, both of which we assume are known. In practice, $G$ can be estimated using data on past experiments, using methods described in Sec. (ref); see also Section (ref). \\
The allocation problem thus reduces to
We show that the problem can be solved in general with simple dynamic programming. Denote by $F(I,N)$ the maximum such value. Notice that
which suggests an efficient approach to calculate $F(I,N)$ by tabulating values $F(i,n)$ for all $i=1,\dots,I, n = 1,\dots,N$. The base cases are simple; when $I = 1$, we just return the value $f(N)$. When $N = 1$, we return the value $f(1) + (I-1)f(0).$ The dynamic programming algorithm takes $O(N^2 I)$ time after the production function is computed. Computing the optimal solution can be done by backtracking, keeping track of the index $j$ which maximizes $F(I,N)$ in equation (ref). If $N$ is prohibitively large, we can consider a minimum experiment cohort size $c_0$ and instead use the update
which requires $O((N/c_0)^2 I)$ time to solve. These methods can be used to solve many related extensions of the return-maximizing framework, as we show throughout the paper. Code implementing the dynamic programming methods may be found on \href{https://github.com/tsudijon/OptimizeExperimentPrograms}{Github}. The repository contains notebooks to replicate all figures in the paper and also an example dataset of treatment effects from a Netflix experimentation program, scaled by a random factor to preserve confidentiality.
At many online technology firms, the traditional framework of $p$-value significance is used; tests are rejected at the $\alpha = 0.05$ level using covariated-adjusted causal estimators, with sample size allocated by power calculations. One-sided $p$-values for the null hypothesis $H_0: \Delta_i \leq 0$ are used to determine whether idea $i$ should be shipped. In the model described in Section (ref), the one-sided $p$-value is given by \[ p_i := 1-\Phi(\widehat{\Delta}_i\sqrt{n_i}/\sigma). \] Perhaps surprisingly, decisions based on this one-sided $p$-value are equivalent to the optimal decisions in the A/B Testing Problem based on the posterior expectation.
A proof of this result and all others can be found in the Appendix. Prop. (ref) suggests there is no loss of generality in considering a $p$-value decision framework versus one which ships when the conditional expectation is positive. Only the $\alpha$ budget per test or equivalently the $t$-statistic threshold needs to be calibrated.
As a result, organizations can leverage existing infrastructure which already calculates $p$-values. Further, the mapping between the A/B Testing Problem and $p$-values suggests an optimal choice for $p$-value thresholds that maximize returns. We explore this further in Section (ref).
The dynamic programming solution suggests that the optimal allocation distributes units equally across some subset of tests $I_0 \leq I$ and zero on the rest. This suggests a useful approximation for the optimal value of the A/B testing problem, given by
where $f$ is the production function in the base A/B testing problem. We will refer to this as the metaproduction function, representing expected program-level returns using the optimal allocation and decision procedure. Studying $F$ lets us understand the sensitivity of program returns to $I,N$ and various features of the prior. For example, consider the idea meta-production function $F_1(I) = F(I,N)$ for a fixed large $N$. The shape of $F_1$ sheds insight into the value of idea generation, which can inform investment decisions into the experimentation program.
To gain intuition on $F$, we will focus on the setting where $G$ is Gaussian with $u(x) = x.$ By conjugacy, the production function can be explicitly calculated. This formula was first written in raiffa2000applied and recently studied in azevedo2023b; similar formulas appear in abadie2023estimating.
This can be simplified to $\mu G\left(\frac{\mu}{\tau^2}\sqrt{\tau^2 + \frac{\sigma^2}{n}} \right)$ where $G(x) = \phi(x) / x + \Phi(x).$ It is shown in azevedo2023b that the production function is generically convex on some interval $[0,\hat{x}]$ and concave afterwards. The value $\hat{x}$ depends on $\mu,\tau$. The following theorem describes $F$ for a general Gaussian prior.
Theorem (ref) yields important insights about the value of additional members and ideas for experimentation programs; implications are given in the following corollaries and in Section (ref). The proof also shows that $x^* \geq \hat x$. Furthermore, the theorem only uses the fact that $f(x)$ is convex for small $x$ then concave for large $x$; if this holds for other priors, then the same results follow.
Understanding the production function also leads to bounds on a type of regret for the base A/B testing problem. This is similar to results quantifying the value of experimentation in abadie2023estimating.
Thus, the loss scales with the number of ideas, and the loss per idea depends on the ratio $\kappa$. Since $\lim_{\kappa \rightarrow \infty} f(\kappa) = \mathbb{E}[\Delta_i^+],$ for large enough $N$ compared to $I$, the loss per idea is negligible.
Finally, we analyze how the changes to the prior density of ideas affects $F$. This shows the value of changing the distribution of ideas as a lever for optimization experimentation returns and extends "comparative statics" results found in azevedo2023b.
A quick inspection of Eq. (ref) shows that $f(n) \rightarrow \infty$ as $\tau \rightarrow \infty.$ Thus if $\tau$ increases quickly enough, even as $\mu \downarrow \infty$, it is possible to obtain more value from an experimentation program.
In many firms, experimentation programs are synonymous with business departments; the managers of these business departments use experimentation as a tool for decision making and optimizing the growth of the program. We highlight the implications of the return-aware framework by answering the following questions that these managers often have in practice:
We will answer these questions under the assumption of no costs. Section (ref) discusses the role of costs as well as risk-aversion, adding significant nuance to the takeaways of the return-aware framework.
This section addresses Question 1. Figure (ref) shows the value of testing $I$ ideas with a fixed allocation pool of 250 million units, split equally amongst the ideas. The $x$-axis spans 4 orders of magnitude; the red dashed line shows the optimum of the curve, and the grey dashed line shows roughly the scale of testing in current practice. With the available allocation and a much larger pool of untested ideas $I$, it would be optimal to run several orders of magnitude more tests of typical quality than current practice. For any value of $I$ to the left of the optimum, Figure (ref) also suggests that we should test all ideas with equal allocation. The resulting test sizes are much smaller than current test sizes, which are on the order of several million and obtained via power calculations.
In this costless regime, many more tests should be run at the expense of higher allocation and power, which is the strategy termed lean experimentation in azevedo2020b, azevedo2023b. The surprising finding is that the strategy is optimal for a wide range of parameters, even with a “light-tailed" Gaussian prior, which gives the opposite intuition from azevedo2020b; this nuance was pointed out in azevedo2023b. Interestingly, lean experimentation is closely related to the paradox of power described in schmit2019optimal. schmit2019optimal considers a sequential model of testing where the goal is to find discoveries as quickly as possible; the optimal strategy is close to running many short, low-powered tests in sequence.
This section answers Question 2 by studying the optimal decision rule from Lemma (ref). Throughout this section, we will consider decision rules for a test with allocation size $5$ million and assume treatment effects come from the Gaussian prior estimated for Program 1 of Figure (ref). Figure (ref) displays several statistics on the optimal decision rule for a particular experimentation program of interest. A striking feature is that the $p$-value threshold is closer to $0.5$ than to $0.05.$ See Section (ref) for a discussion on minimax approaches that suggest similar $p$-value thresholds. This recommendation aligns with several other papers, which also show that $p < 0.05$ decision rules are too conservative goldberg2017decision, azevedo2020b.
We will reinforce this finding by demonstrating the suboptimality of $p < 0.05$ decision results in various settings. Given a linear utility with no costs, one way to demonstrate subtoptimality is to visualize other Gaussian priors that induce an optimal decision rule where $p < 0.05$. By Equation (ref), we display several priors whose parameters $\mu,\sigma$ satisfy $-\mu \sigma/\sqrt{n} = \Phi^{-1}\left( 0.95\right) \tau^2.$ Figure (ref) shows several such distributions, superimposed on the actual estimated prior. Using the $p < 0.05$ rule corresponds to an unnecessarily pessimistic prior where there is little mass above $x > 0$. The results also show that the general recommendation of adopting more conservative decision thresholds is robust to mis-estimation of the prior distribution.
We may also calculate the average return lost when using the suboptimal $p < 0.05$ decision rule. Figure (ref) compares two decision rules, (1) shipping when $\mathbb{E}[\Delta | \widehat{\Delta}] > 0$ and (2) shipping when $p < 0.05$, using the prior $G.$ In the top plot, we compare the expected return from optimally allocating $N$ units to $I$ ideas, with both decision rules. For (1), this is exactly the metaproduction function, while for (2), we obtain the analog of the metaproduction function, using instead $f_{p < 0.05}(n) = \mathbb{E}\left[\Delta_i\mathbf{1}\left\lbrace \widehat{\Delta}_i \geq \frac{1.96\sigma}{\sqrt{n}} \right\rbrace\right].$ The simulated example shows that using $p < 0.05$ decisions can miss out on nearly 80% of returns, as compared to the optimal decision strategy, depending on $I,N$. The bottom plot compares the production function $f(N)$ and $f_{p < 0.05}(N).$
Studying the metaproduction function for an experimentation program answers Question 3. Figure (ref) shows the metaproduction function with estimated Gaussian parameters on Program 1 of Figure (ref). The results show that increasing the allocation pool has rapidly diminishing returns, while adding more untested ideas into the pool remains valuable, with slowly diminishing marginal returns. The theory presented in Section (ref) suggests that beyond a point, adding additional ideas to the pool does not increase program-level returns. The scale of current practice is nowhere near this breakpoint.
The results of Section (ref) suggest the need to generating many more ideas, given that idea quality remains the same. The right panel of Figure (ref) demonstrates the value of doing so, with a fixed allocation pool of 250 million units. With additional information on the costs of idea generation, we may use the metaproduction function $F$ to determine the size of investment into the program, in terms of the number of untested ideas available.
The theory in Section (ref) also suggests a few insights. The first is intuitive: organizations should increase idea average quality, as Corollary (ref) shows. If this is difficult, an alternative is to increase the diversity of ideas, corresponding to increasing $\tau$. Secondly, Theorem (ref) suggests under which conditions the lean experimentation is optimal: organizations should test all ideas in the pool if $I \leq N/x^*$ where $x^*$ is the global maximum of $f(x)/x$ where $f$ is the production function of the program.
Our discussion of optimizing returns has only touched on the role of costs. If the implementation cost for idea $i$ is $s_i$ and the testing cost is $t_i(n_i)$, as a function of the allocation $n_i$, then the production function changes and is given by \[ \mathbb{E}\left[(\mathbb{E}[\Delta_i | \widehat{\Delta}_i; n_i] - s_i)^+ \right] - t_i(n_i). \] Recall that the production function measures the expected return from allocating $n_i$ units to test a typical idea $i$, using the optimal decision. Here, the optimal strategy is to ship all ideas with posterior expectation greater than the implementation cost $s_i$. The allocation problem reduces to maximizing a separable objective with a simplex constraint, so our computational techniques still apply.
The difficulty with understanding the role of costs is that costs are typically difficult to measure. For example, there are hidden costs per test that present as accumulated complexity, when more and more treatments are incorporated into the platform. Measuring costs of human labor for collaborative projects is not straightforward. Furthermore, the metric of returns in most experiments is different from dollars, and converting between the two metrics often relies on a noisy black-box. As a result, the exact takeaway for experimentation practice depends on the firm. We suggest a suite of several exercises that firms can use to study decision procedures given their bespoke costs and risk tolerances, as follows. The exercises are illustrated on Program 1 of Figure (ref). See also azevedo2020b for an excellent related discussion on data from Microsoft Bing.
We first study the dependence of $p$-value thresholds on implementation costs; we can reverse-engineer the implementation costs for which $p < 0.05$ is optimal given a linear utility and the estimated prior from Figure (ref). With cost $s_i$, it is optimal to ship a tested idea $i$ whenever \[ \mathbb{E}[\Delta_i | \widehat{\Delta}_i] \geq s_i. \] By the same arguments as Prop. (ref), there is a unique value of $\widehat{\Delta}_i$ above which it is optimal to ship. Figure (ref) shows the implied $p$-value threshold as a function of $s_i$. The result suggests that the $p = 0.05$ decision threshold is optimal with implementation costs of about 9 times the magnitude of $|\mu|$. Whether this cost is significant or insignificant greatly depends on the scale of $|\mu|$ and other parameters specific to the experimentation program. Interpreting this cost should be done in a firm-specific manner. However, rough conversions between this metric and dollars in the Netflix context suggests an implementation cost equivalent to a year's worth of work for several engineers or data scientists. This is realistic for some ideas, but not others.
Along similar lines, we can visualize several utility functions for which $p < 0.05$ is optimal. Consider $u(x) = x + bx\mathbf{1}\left\lbrace x < 0 \right\rbrace, b \geq 0.$ $b = 0$ recovers the linear utility we have primarily focused on. The optimal decision given this utility is to ship all tests for which
By normal conjugacy, $\Delta_i | \widehat{\Delta}_i \sim \textsf{N}(m,s^2)$ with posterior mean and variance given in Example (ref). Using integration by parts, (ref) reduces to solving for the value of $c$ which satisfies \[ m\left( 1 + b\Phi\left(-\frac{m}{s}\right)\right) - bs\phi\left(\frac{m}{s}\right) = 0. \]
Figure (ref) displays the resulting optimal $p$-value thresholds for various risk averse utilities with different choices of $b$, for Program 1 of Fig. (ref). The results suggest that a $p < 0.05$ decision is optimal for a risk-averse utility which weights losses about 60 times more than gains. The exact multiple varies depending on the properties of the prior distribution. When repeated for the other priors in Figure (ref), the implied risk-aversion still weights losses much more heavily than gains.
Finally, Figure (ref) shows the impact of fixed costs per test on the optimal allocation in various scenarios. The same curve as in Figure (ref) is pictured in blue, while the other curves show the cumulative returns to testing $I$ ideas, given a fixed testing cost $t_i(n_i) = c$. We test four settings, corresponding to no cost, an unspecified economically significant cost $c$, and multiples $5c, 10c$. In each of these settings, the optimal allocation is several times larger or even orders of magnitude larger than current practice at Netflix.
The results of these exercises will result in different takeaways per firm. However, it is clear that when framing experimentation in the return-aware framework, one should both measure and incorporate costs into any practical analysis. Further research should be conducted on the sensitivity of these results to costs.
We also highlight the need for better infrastructure to measure costs and drive them down as much as possible. The current state of experimentation in online platforms generally suggests that costs to testing are rapidly decreasing. At many online technology firms, there are centralized teams -- experimentation platforms -- whose explicit mandate is to reduce the friction of running A/B tests; for smaller organizations without such teams, there are external vendors who provide experimentation services. One ultimate goal of these efforts is to bring experimentation into the costless regime, where the return-aware framework suggests significant changes to current practice. For example, in experimentation platforms, a common business question is to understand the value of investing a large capital investment to decrease fixed costs for future experiments. The return-aware framework can be used to quantify tradeoffs in case studies like these.
At many firms, experimentation is conducted in multiple areas which have different priors. Multiple experimentation programs may share constrained resources, including allocation or R&D resources. Addressing Question 4, we discuss an extension of the A/B testing problem to answer how allocation should be divided amongst the programs, in order to optimize returns. Answering this question provides useful insight for making investment decisions across multiple experimentation areas.
Suppose that there are $P$ experimentation programs indexed by $p=1,\dots,P$, each with $I_p$ tests, indexed by $i=1,\dots,I_p$. Assume that test returns are drawn from distributions $G_p$ so that
The return-aware framework is the same, except unit allocation to each program comes from a shared pool of $N$ units. This variant of the framework can be analyzed in the same way, resulting in the optimization problem
Understanding the solution of this problem requires us to calculate the maximum value of the previous subproblem for different values of $N.$ Let $N_p = \sum_{i=1}^{I_p} n_{i,p}$ be the number of individuals allocated to program $p$. Then the optimization problem above may be written as
where $F_p(N_p)$ is the solution of the A/B Testing Problem for program $p$, with prior $G_p$ and maximum allocation $N_p$. We may compute every value of $F_p(N_p)$ using the dynamic programming solution to the base A/B testing problem. Letting $\mathcal{M}(a,b)$ denote the solution to the problem above with $a$ programs and $b$ total units, dynamic programming can be used to solve (ref), with the update \[ \mathcal{M}(a,b) = \max_{j=0}^b \left(\mathcal{M}(a-1,b-j) + F_a(j) \right) \] and base case
Figure (ref) shows the metaproduction functions for the same three different Netflix experimentation programs in Figure (ref). The number of ideas for each program roughly reflects the relative sizes of each program in past data. The shape of the metaproduction functions immediately shows that we should not expect the optimal allocation to place all mass on one program. Solving the resulting dynamic programming gave an allocation of $[112, \ 89, \ 49]$ million units to the three experimentation programs respectively.
\paragraph*{Shared R&D Resources.} A similar approach can answer questions on managing experimentation programs which share resources for generating ideas. Suppose we have $P$ different experimentation programs, which do not share allocation pools; within each program, tests have disjoint allocations. Let each program $p$ have access to an allocation pool of size $N_p$. Suppose further that the cost to create ideas is the same across programs, and there is budget to create $I$ additional ideas across all $P$ programs.
The problem is to choose the number of ideas $I_p$ for each program $p$, the allocation $(n_{i,p})_{i=1,\dots,I_p}$ to test ideas for program $p$, and the subset $S_p$ of ideas to ship, in order to maximize cumulative return from all experimentation programs. This yields the optimization problem:
The solution of this problem shares the dynamic programming structure of the heterogeneous A/B testing problem. Let $F_p(I,N_p)$ be the optimal value of the base A/B testing problem with $I$ ideas to test and $N_p$ units. Then by similar arguments to the previous case, the optimization problem may be written as
It is also possible to weight each program differently in the objective, which may be useful if the programs target different metrics. Again, dynamic programming may be used to solve this. Fixing the total number of units $N$, let $\mathcal{M}(I,P)$ be the maximal value of the program (ref). By casework on the value of $I_p,$ the maximal value satisfies the equality \[ \mathcal{M}(I,P) = \max_{j=0}^I \left( \mathcal{M}(I-j, P-1) + F_p(I, N_p)\right). \] The optimal allocation can be rediscovered by keeping track of the value $j$ which maximizes the quantity above. Figure (ref) shows the solution of the resource allocation problem for the same three programs as Figure (ref). Amongst three programs, the allocation is $[30,20,0]$. Although the solution suggests zero allocation should be given to Program $3$, the results would change if the allocation pool for the program were much larger, or if the prior were to change.
The goal of optimizing returns can be formalized by models other than the A/B Testing Problem in Section (ref). For example, experimentation programs may enroll units in multiple tests at once, or may select at most one idea to be implemented, if the treatments are mutually incompatible. In another direction, a new experimentation program may not have enough data to estimate a prior $G$, raising the need for alternative decision procedures. In this section, we explore the resulting optimization problems and decision theory for these settings.
Returning to the constraint $\sum_i n_i = N$, a more realistic constraint in many experimentation programs is to allow for units to be enrolled in multiple tests at once. Assuming that units are exchangeable for every test in an experimentation program and every unit can be enrolled at most $k$ times, one can solve the A/B Testing Problem by imposing the constraints $\sum_i n_i = kN, n_i \leq N$. Whenever these constraints are satisfied, there is an allocation of units to tests such that every unit is enrolled in at most $k$ tests, each test has $k$ units, and units cannot be enrolled multiple times in the same test. The dynamic program can be solved in a similar fashion using now the update \[ \mathcal{M}(i,n) = \max_{j=0}^{\min(n,N)} \left( \mathcal{M}(i-1, n-j) + f(j) \right) \] and evaluating $\mathcal{M}(I,kN).$
In practice, multiple experimentation programs share some allocation pools but not others, suggesting a different set of constraints in the problem. Suppose we have the following data:
Let $f_p(n)$ be the production function for the $p$th program. Maximizing returns subject to the constraints above can be formulated as the optimization problem
Under a Gaussian prior, let $\hat{x}_p$ be the threshold above which $f_p$ is concave. If we replace the constraint $n_i^{(p)} \geq 0$ by $n_i^{(p)} \geq \hat{x}_p$, the optimization problem (ref) becomes a convex optimization problem. If $\hat{x}_p$ is small enough, the convex optimization proxy is a good approximation to problem (ref); one can round down $n_i^{(p)}$ to zero to avoid running tests with a very small number of units, for operational reasons.
Consider a mutually exclusive version of the A/B testing problem, where only one test may be shipped amongst the pool of tested ideas. azevedo2020b study this problem as Extension 2 of their framework, in the Supplemental Information G.2 section of their paper. The optimal test $s$ to ship is given by
\[ \operatorname*{argmax}_{i=1,\dots,I} \; \mathbb{E}[\Delta_i | \widehat{\Delta}_i] \]
if the largest posterior expectation is positive, which is recorded in SI G.2 of azevedo2020b. The mutually exclusive A/B testing problem with this objective reduces to the optimization problem.
By considering symmetric solutions which equally allocate across some number of tests, it suffices to choose $I_0 \leq I$ which maximizes
Figure (ref) shows a plot of (ref) as a function of $I_0$ for various values of $N$. In this case, the optimal number of tests to run is much smaller than in Section (ref).
The expression is difficult to compute analytically, but we may provide useful approximations. Notice that $\max_{i=1,\dots,I_0} \mathbb{E}[\Delta_i | \widehat{\mathbf{\Delta}}; N/I_0]$ is equal to \[ \frac{\frac{\sigma^2 I_0}{N}}{\tau^2 + \frac{\sigma^2 I_0}{N}}\mu + \frac{\tau^2}{\tau^2 + \frac{\sigma^2 I_0}{N}} \max_{i=1,\dots,I_0} \widehat{\Delta}_i. \] \sloppy By well-known arguments chatterjee2014superconcentration for the concentration of maxima of Gaussian random variables, this suggests that $\max_{i=1,\dots,I_0} \mathbb{E}[\Delta_i | \widehat{\mathbf{\Delta}}; N/I_0]$ is exponentially concentrated around
with fluctuations of order $1/\sqrt{2\left(\tau^2 + \frac{\sigma^2 I_0}{N}\right)\log I_0}$. Whenever the mean (ref) is larger than several multiples of the fluctuation, we should expect that Eq. (ref) is approximately equal to \[ \mathbb{E}\left[\max_{i=1,\dots,I_0} \mathbb{E}[\Delta_i | \widehat{\mathbf{\Delta}}; N/I_0] \right] = \mu + \sqrt{\frac{2\log I_0}{\tau^2 + \frac{\sigma^2 I_0}{N}}}. \] The approximation (ref) can be used to inform the optimum value of $I_0 \leq I$ which maximizes expected return. Optimizing this in $I_0$, we obtain an approximation to the solution of the mutually exclusive A/B testing problem.
For newly launched experimentation programs, there is little data to estimate a prior $G.$ To navigate around issues of choosing a subjective prior, the goal of maximizing welfare $\sum_{i \in S} \Delta_i$ can be recast as finding the minimax decision for the loss \[ \sum_{i} \Delta_i^+ - \sum_{i \in S} \Delta_i. \] Suppose for each $i$ we observe $X_i \sim \textsf{N}(\Delta_i, \sigma^2/n_i)$ with fixed $n_i.$ The minimax decision, perhaps surprisingly, is to ship all $X_i > 0,$ which is a much less conservative decision than $p < 0.05$. The resulting decision is similar to the conclusions found in Section (ref).
In the single test case, this result is well-known in the statistical treatment literature in economics; see manski2004statistical, stoye2009minimax, and further references in manski2019treatment, which focuses on minimax regret problems. The result extends simply to multiple observations, with different variances, for which we provide a simple proof. See also chiong2023minimax, joo2023getting for a more refined discussion of minimax decisions in A/B testing.
For a single test, Section 4 of azevedo2023b derives the same expression for minimax risk. The expression is used to size a single experiment given external costs to enrolling members. We show how the same expression suggests a strategy for optimizing returns from a portfolio of A/B tests, suggesting the lean experimentation is optimal from a minimax perspective.
An empirical Bayesian approach is also natural here, as in guo2020empirical. If we are willing to assume $\Delta_i \sim G$ for an unknown prior which cannot be estimated from past data, the problem naturally becomes an empirical Bayes problem. Empirical Bayes for welfare maximization does not seem to have been studied explicitly, although some results on the nonparametric maximum likelihood procedure (NPMLE) can be applied; see Equation 3.7 of chen2022empirical. It is unclear whether there is a matching regret lower bound and whether the plug-in NPMLE approach is optimal. We raise this as an interesting direction for future research.
Given a prior on treatment effects and the mandate to optimize returns, organizations conducting experimentation are typically better served by a return-aware framework rather than null hypothesis statistical testing. As a consequence -- with no costs -- organizations should be testing many more ideas, lowering the barriers to testing as much as possible, and reconsidering decision thresholds for shipping tests.
There are several reasons why the $p < 0.05$ paradigm remains entrenched in digital platforms. An appropriate combination of inertia, loss aversion, costs, and prior distribution could make conservative thresholds, in fact, optimal. However, running the exercises outlined in Section (ref) for Netflix experimentation programs, we find that individually varying each of these factors suggests either an unrealistic prior, an unusual degree of risk-aversion, or overly large implied costs. Although the results will change per firm, in our experience, many firms do not justify current practice with informed cost measurements or decision theoretic considerations. They rather default to statistical tradition.
One widespread argument for conservative decision making is the cost of false positives in A/B tests berman2022false, kohavi2024false. As part of relaxing $p$-value thresholds, the return-aware framework necessarily eschews false positive rate control. Figure (ref) reflects this. It is important to note that most of these false positives have treatment effects of small magnitude.
The primary concern around false positives, even if the magnitude is small, centers around the risk of building future ideas on false positive discoveries, which may result in a sub-optimal path of innovation relative to a counterfactual based on a true discovery. But this appears to confuse decision making and epistemology: one can imagine a simple framework where we make decisions using a return-optimization framework, but gain knowledge about the platform and population using results which are statistically significant. That is, we only invest in ideas which pass a higher bar of statistical evidence. Overall, there appears to be a lack of empirical or theoretical study for the impact of false positives on future innovation; we raise this as an interesting direction for future research.
Of course, the return-aware framework should not always be applied. For experiments in practice which are designed to learn about the population of interest, a null hypothesis testing framework with a conservative decision threshold applies. For atypical experiments that are not plausibly drawn from $G$, one can default to a minimax framework as in Section (ref) or the traditional hypothesis testing framework.
We also emphasize the importance of choosing the right metric to optimize in an experimentation program. Optimizing short-term metrics in the online technology industry runs into the risk of creating negative externalities for users, and potential backlash. In this way, the framework pairs well with recent research on surrogates for long-term outcomes.
Finally, we note that the return-aware framework is potentially useful in other settings beyond A/B Testing. For example, experimentation teams in political campaigns desire to maximize donations or voter turnout. Economic field experiments can be used in service of maximizing social welfare. Agricultural experiments aim to maximize crop yields. In the first two settings, there is an important additional nuance in the literature, which focuses on the generalization of the sample ATE to the population ATE, as well as a focus on policy learning. See for example manski2004statistical, kitagawa2018should, athey2021policy, ben2022policy and the references therein; some of the works focus on the observational case, which is different from our setting. See also milkman2021megastudies for an experimentation program in behavioral science with multiple parallel experiments, where our setting may apply.
In the remainder of this section, we conclude with a list of directions for future research.
\paragraph*{Estimation of $G$.} We focus primarily on the problem of optimizing future test design, having knowledge of $G$. Under a fully Bayesian specification, multi-level hierarchical models such as ejdemyr2023estimating can be used to obtain the posterior on $G$ given past data on A/B tests. From a frequentist perspective, $G$ needs to be estimated from past data; we opt for this approach, using maximum likelihood estimation for a simple normal parametric family. azevedo2020b uses maximum likelihood for a $t$-distributed parametric family with shape and location parameters; see azevedo2019empirical for an overview of methods to estimate $G$.
There are several interesting statistical questions in the estimation of $G$, since the quantity of interest we primarily care about is the production function $f(n).$ Firstly, one should understand the risk of incorrectly estimating $G$ from past data, under a parametric or nonparametric model. The estimated tail behavior is important, based on the reasoning in azevedo2020b. Secondly, understanding the value of non-parametric approaches, such as the NPMLE, is important. Although the NPMLE learns priors which are atomic, it would be interesting to investigate to what extent the NPMLE can be used to learn the production function $f(n)$. This avenue is promising, as the NPMLE (e.g. koenker2014convex) has several adaptivity properties that make it suitable to learning both light-tailed and heavy-tailed priors shen2022empirical, soloff2024multivariate, jiang2020general, although results on estimation rates for $G$ are still currently being researched soloff2024multivariate. Notice by Tweedie's formula that the production function may be written as
where $m_{\hat{\Delta},\sigma^2/n}(x)$ is the marginal density of an observation $\hat{\Delta}_i$ with variance $\sigma^2/n.$ Eq. (ref) appears to be useful in obtaining error bounds when estimating $G$ by NPMLE, using the results of jiang2009general, jiang2020general, soloff2024multivariate, or by other estimates guo2020empirical. Deconvolution kernel estimators fan1991optimal, delaigle2008density, meister2009density may also be used here, although they require tuning parameters for the heteroskedastic problems in our setting. NPMLE, on the other hand, is tuning-free.
In estimating $G$, we ignore the multi-arm structure of most experiments in practice. This leaves a lot of data on the table. Estimation of the prior $G$ with multi-arm structures can be done with a two-level hierarchical model; techniques such as the NPMLE can be used here, or fully Bayesian models such as ejdemyr2023estimating may be used. After estimating $G$, there is the natural question of how to optimize returns from a portfolio of multi-cell experiments.
\paragraph*{Sequential Frameworks.} The A/B Testing problem is a one-shot optimization. In practice, we have multiple time periods to space out tests. Incorporating time into the problem suggests several tradeoffs. For example, there would be a tradeoff between running every idea with high powered experiments, taking a long time, versus testing all ideas at once with very under-powered tests. Here is one model which captures some of aspects of these tradeoffs. Suppose that we have $t = 1,\dots,T$ time periods, $I$ total ideas to test, and $N$ units. During a time period $t$, we may select a number of ideas to test using $N$ units; for a given period, let $I_t$ be the number of ideas tested. The problem is to choose an allocation $(n_{i,t})_{i=1,\dots,I_t}$ to test ideas at time $t$, the idea allocations $I_t$, and the subset $S_t$ of ideas to ship, in order to maximize cumulative return by a fixed time, for any choice of weights $w_t$:
The problem may be solved using dynamic programming in a fashion similar to the problems in Section (ref). Figure (ref) shows the solution of the sequential A/B testing problem. Such insights are useful for planning and scheduling exercises around experimentation. For a metric which upweights ideas that are shipped earlier, it is intuitive that more tests are shipped early on. As $T$ increases, one should expect the allocation of tests over time to even out.
The sequential version of the A/B testing problem we introduce allows managers to answer scheduling questions. However, it misses an important aspect of A/B test decision making - test lengths are variable, and there is an option of waiting longer to collect more information on a treatment. Sequential decision making for a single decision has been studied by Wald and collaborators wald1992sequential, wald1950bayes, arrow1949bayes, and has seen intense resurgence in the context of digital platforms maharaj2023anytime, ham2022design, johari2022always. The problem with multi-arm tests is closely related to the multi-armed bandits, best-arm identification, and sequential clinical trials literatures, depending on the loss function specified. See kasy2021adaptive, adusumilli2021risk, adusumilli2022neyman, adusumilli2022sample and the literature reviews within. From the perspective of experimentation programs, there is the interesting extension of how to manage a portfolio of A/B tests sequentially. Under the assumption of no cross-allocation, at each time one must choose an allocation of units, and for each test, there is a decision to either ship the idea or continue testing. Deriving optimal strategies to maximize cumulative returns would be of clear interest.
\paragraph*{Data-driven Program Groupings.} Operationally, it is easy to define collections of A/B tests which should naturally be grouped together. But this process is easily criticized - are there other groupings of A/B tests for which the assumption of i.i.d. draws is more realistic? This is the relevance problem of empirical Bayes efron2012large, which is also highlighted in the conclusion of deng2015objective. Experimental metadata could be used as covariates to inform a “data-driven" grouping. See ignatiadis2019covariate for a related approach.
\paragraph*{Modelling Idea Generation.} In the setup considered so far, the firm has $I$ ideas to test and the generative process for arriving at these ideas is left unmodelled. As hinted earlier, colloquial discussions about the prevalence of conservative launch rules are motivated by seeking to avoid false-discovery “trajectories". Overall, the cost of false positives on the rate of future innovation appears to be understudied. Is there empirical evidence or simple theoretical models to support this phenomenon? Simple dynamic models of innovation such as callander2011searching could be used to study extensions of the A/B testing problem with this issue in mind.
Furthermore, the current work abstracts from the distribution of ideal quality when testing an order-of-magnitude larger number of ideas; it may plausibly be the case that the average quality of each additional idea may diminish in certain experimentation programs, which may provide important additional considerations in practice.
\paragraph*{Diminishing Returns and Data Relevance.} Industry wisdom suggests that for an optimized product, most ideas should have small impact. For relatively unoptimized products, many ideas result in big wins. Thus, older data should be less relevant for estimating a prior; thus points the need for research on how best to weight past data under slow distribution shift.
Moreover, if the distribution $G$ is re-estimated in the future, there is a concern that allocation which optimizes returns to experimentation is at odds with the allocation that best allows the manager to estimate $G.$ In the setting with a normal prior, what is the optimal allocation to best estimate $\mu,\sigma^2$? Are a few high-powered experiments better than many low-powered experiments for estimating the prior? Consider allocations which equally split $N$ units across some fixed number of tests $I_0$, and zero on the others. Then the estimates $\widehat{\Delta}_i$ are i.i.d from $\textsf{N}\left(\mu,\tau^2 + \frac{\sigma^2 I_0}{N}\right)$; the MLE estimates for $\mu,\tau^2$ reduce to the sample mean and sample variance, so that
Thus $\operatorname*{Var}(\hat{\mu}_{\textsf{MLE}})$ is strictly decreasing in $I_0$, and $\operatorname*{Var}(\hat{\tau}^2_{\textsf{MLE}})$ is minimized when $I_0 \approx \frac{\tau^2N}{\sigma^2}$. The optimal $I_0$ values for either the standard deviation and variance were typically much larger than the number of tests currently being run at Netflix. As a result, increasing the number of experiments being run optimizes both returns to experimentation and the ability to learn the prior $G$. This computation suggests similar tradeoffs for more interesting estimators, which could be a direction for future research.
We thank Eduardo Azevedo, Travis Brooks, Michael Howes, Michael Lindon, Tomoya Sasaki, and attendees of CODE@MIT 2024 for helpful comments and discussion.