Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
84,911 characters · 15 sections · 65 citation commands
{\it Keywords:} Experimental designs, A/B testing, inventory management, interference.
Across modern digital platforms, pharmaceutical innovation, and empirical economics, researchers rely on controlled experiments, such as A/B testing, clinical trials, and randomized controlled trials (RCTs), to rigorously evaluate the effectiveness of new strategies or interventions. The fundamental principle of traditional randomized experiments involves randomly assigning experimental units to treatment and control groups, then inferring the effect of intervention by measuring the average performance difference between groups on metrics of interest. In practical applications, decision makers employ A/B testing to evaluate whether implementing a new intervention (treatment) across the entire platform would yield better performance compared to the existing approach (control), referred to as Global Treatment Effect (GTE).
On large online platforms, thousands of experiments are conducted daily to assess the impact of changes, ranging from minor adjustments to major overhauls. Extensive literature exists on bias analysis across different experimental designs and platforms, including examples from room-sharing platforms johari2022experimental, advertising platforms basse2016randomization,liao2024interference, recommendation systems si2023tackling, farias2023correcting, zhan2024estimating, ride-sharing platforms bright2024reducing, and gaming platforms weng2024experimental.
While the technology industry has widely embraced A/B testing, such data-driven methods remain underexplored and underutilized in other industries, such as inventory management. However, as Prof. Thomke stated in his book Experimentation Works: The Surprising Power of Business Experiments thomke2020experimentation:
We also anticipate that A/B testing will increasingly be adopted in those industries as well. In particular, industries such as inventory management have witnessed a prominent trend to become more and more data-oriented, providing an advantageous position to effectively implement A/B tests.
This paper studies A/B testing for inventory management. In recent years, a growing body of work has proposed automatic, data-driven inventory control methods shi2016nonparametric,lyu2024minibatch,xie2024vc. While these methods exhibit strong theoretical guarantees under stylized assumptions, how to reliably evaluate their practical performance in real-world environments—where such assumptions are frequently violated—remains largely underexplored. The dominant evaluation paradigm relies on simulation: practitioners estimate demand distributions from historical data and construct behavioral models to capture customer responses after stockouts. However, this approach suffers from several fundamental limitations. First, inventory data are intrinsically censored: demand after a stockout is unobservable, yet this latent demand is often critical for assessing the true performance of an inventory policy. Second, customer behavior following stockouts—such as substitution, abandonment, or delayed purchase—is complex and context-dependent, making it difficult to model accurately and robustly. Third, high-fidelity, large-scale simulations are computationally expensive and operationally burdensome, especially in multi-item, multi-period settings. These challenges cast doubt on the reliability and scalability of simulation-based evaluation.
Motivated by these limitations, this study investigates A/B testing strategies for evaluating inventory control policies and provides guidance for experimental design across different operational regimes. Specifically, we focus on A/B testing different inventory policies in multi-item, multi-period systems with lost sales and a warehouse capacity constraint snyder2019fundamentals. We analyze two canonical experimental designs: switchback experiments cochran1941double,bojinov2023design,xiong2023data and item-level randomization. In switchback experiments, the treatment assignment for all experimental items randomly alternates between the control group and the treatment group over time. In contrast, item-level randomization involves randomly assigning items to treatment and control groups, with their treatment assignments remaining fixed throughout the experiment.
However, both the aforementioned experimental designs have limitations due to bias caused by interference. In switchback experiments, temporal interference or carryover effects are present, meaning that a treatment not only impacts the outcome of the current period but also influences the system’s state and subsequent outcomes in future periods. For item-level randomization, cannibalization effects may occur due to capacity constraints, where the treatment assigned to one item affects the outcomes of other items. These scenarios violate the Stable Unit Treatment Value Assumption (SUTVA) in causal inference imbens2015causal. And previous studies showed that the bias caused by interference can be as large as the GTE itself blake2014marketplace,fradkin2019simulation,holtz2020reducing.
In this work, we examine the bias of the naive inverse probability weighting (IPW) estimator under different experimental designs, including switchback experiments and item-level randomized experiments. Our contributions are summarized as follows:
The rest of the paper is organized as follows. In Section (ref) we review related work on A/B testing and multi-item inventory control. Section (ref) specifies the model of multi-item multi-period inventory control, and the inventory policy we use. In Section (ref), we introduce the experimental designs and the treatment effect estimator. In Section (ref), we study A/B testing of inventory policies induced by different demand estimates and characterize the sign of the bias in the estimated global treatment effect under different experimental designs. In Section (ref), we conduct A/B testing experiments based on real-world data to empirically verify our theoretical findings. Finally, we conclude with several practical implications and future work in Section (ref).
Experimental designs and bias analysis. There is extensive research on experimental designs in two-sided marketplaces, such as pouget2019variance, holtz2020reducing, johari2022experimental, li2022interference, bajari2023experimental, dhaouadi2023price, harshaw2023design, bright2024reducing, masoero2024multiple, zhan2024estimating. While our setting also exhibits a two-sided nature—product side and temporal side—it differs fundamentally from the demand and supply sides typically analyzed in two-sided marketplaces. Specifically, our two-sided structure is “asymmetric," whereas the demand and supply sides in traditional marketplaces are generally “symmetric" from the perspective of bias analysis. Furthermore, masoero2023efficient is closely related to our setting, as they also adopt this view. However, we systematically analyze different designs within the context of inventory control.
Switchback experiments and temporal interference have been also widely studied in the literature; see, for example, glynn2020adaptive,hu2022switchback,bojinov2023design,xiong2023data,ni2023design,li2023experimenting,han2024population. And some researchers are using reinforcement learning techniques to tackle this type of interference farias2022markovian,shi2023dynamic,chen2024experimenting. Our work specifically focuses on inventory management, enabling us to characterize carryover effects and perform detailed bias analysis. Additionally, some studies address scenarios with irreversible treatment conditions, where items assigned to the treatment group cannot revert to the control group. In such cases, staggered rollout designs are used, gradually increasing the proportion of individuals in the treatment group. han2023detecting and boyarsky2023modeling developed statistical tests for interference in A/B testing with increasing allocation, while xiong2024optimal analyzed this design and established optimality conditions for both non-adaptive and adaptive settings.
Other types of experimental designs and interference structures have also been studied in the literature. These include clustering randomization and network interference ugander2013graph,aronow2017estimating,pouget2019variance,jagadeesan2020designs,doudchenko2020causal,yu2022estimating,leung2022rate,harshaw2023design,jia2023clustered,candogan2024correlated,shirani2024causal, experimental designs in auctions basse2016randomization,chawla2016b,liao2023statistical, experiments under non-stationarities wu2022non, simchi2023non,wu2024nonstationary and adaptive experimental designs kasy2021adaptive,qin2022adaptivity. For further details, we refer readers to the excellent tutorial by zhao2024experimental.
For A/B testing in inventory systems, recently chen2025b shows that product stockouts create interference across treatment and control groups, causing standard RCT estimators to overestimate treatment effects. Their analysis focuses on bias induced by stockout-driven substitution, whereas our setting highlights how improvements in demand forecasting propagate through inventory decisions and interact with experimental design to generate bias.
Multi-item inventory models. The research of inventory models with multiple items and capacity constraints dates back to $1960$s. veinott1965optimal and ignall1969optimality considered the multi-period model with stationary demand distributions, and focused on conditions under which a myopic multi-item base-stock policy is optimal. nahmias1984efficient and erlebacher2000optimal investigated the multi-item newsvendor problem with capacity constraints, introducing both optimal and heuristic methods for solving it. beyer2001stochastic,beyer2002average considered linear ordering costs and convex backlogging costs, and demonstrated that the base-stock policy is optimal in terms of infinite-horizon discounted costs and long-term average costs, respectively. In additional to a single capacity constraint, ben1993constrained,lau1995multi,decroix1998optimal,downs2001managing,niederhoff2007using,abdel2007quadratic,zhang2010multi considered multiple constraints. Readers are referred to turken2012multi for a detailed review. More recently, shi2016nonparametric and guo2024online studied the multi-item lost-sales inventory system with a warehouse capacity constraint under the lens of demand learning from censored data, and federgruen2023multi,federgruen2023scalable extended this line of research by incorporating additional chance constraints and proposing asymptotically optimal policies with a large number of items.
Learning algorithms for inventory control. Recent research has focused on developing data-driven learning algorithms to estimate (possibly censored) demand functions. A recent paper by xie2024vc provides a theoretical foundation for this learning problem. Learning algorithms in inventory control can generally be categorized into parametric and nonparametric approaches, depending on the assumptions made about the demand distributions. Parametric learning algorithms mainly include the traditional estimate-then-optimize approach liyanage2005practical,bertsimas2020predictive,elmachtoub2022smart and Bayesian learning approach lariviere1999stalking,chen2008dynamic,chu2008solving, araman2009dynamic, wang2017bayesian. For nonparametric learning, Stochastic Gradient Descent (SGD)-based algorithms is a popular approach. burnetas2000adaptive developed a gradient descent type algorithm for perishable inventory system, and huh2009nonparametric first established the regret bound of SGD-based algorithm in single-product inventory systems. Subsequently, agrawal2019learning,zhang2020closing developed different online learning algorithms for the same inventory system with positive lead times, and chen2020optimal,yuan2021marrying,ding2024feature, yang2024nonparametric,lyu2024minibatch developed SGD-based algorithms in other inventory models. Another stream of nonparametric demand learning research focused on the sample average approximation (SAA) approach kleywegt2002sample,besbes2013implications, levi2015data,lin2022data, fan2022sample, qin2023sailing. Other related works include concave adaptive value estimation approaches godfrey2001adaptive,powell2004learning, coupling arguments for regret analysis agrawal2019learning,zhang2020closing,chen2024learning, learning algorithms for joint pricing and inventory control chen2021nonparametric,chen2022dynamic and dynamic pricing keskin2014dynamic,wang2021multimodal,bu2022context. More recently, lyu2024ucb developed an Upper Confidence Bound (UCB)-type learning framework, while guo2024online proposed mirror descent algorithms to address high-dimensional inventory systems. Although these methods exhibit strong theoretical properties under specific assumptions, their practical performance in real-world environments remains largely unexplored. Our paper presents testing methods that provide definitive evaluations of these methods' performance in practice.
{We consider a periodic-review inventory system of $N$ different items with a shared warehouse capacity $B$ over a planning horizon of $T$ periods. We consider a lost-sales inventory model with zero replenishment lead time; that is, unmet demand is lost. The zero lead time assumption is made for tractability, as the lost-sales problem with positive lead times is known to be notoriously complex, even for a single product (see zipkin2008structure). The true demand distributions are unknown, but the decision maker can have estimates of those distributions (see the details in Section (ref)), and make inventory decisions accordingly. }
We denote the random demand vector in period $t\in[T]$ by $\mathbf{D}_t=(D_{1,t},...,D_{N,t})$, where $D_{n,t}$ represents the demand of item $n\in[N]$ in period $t$. The cumulative distribution function (CDF) of $D_{n,t}$ is denoted by $F_{n,t}$.
In each period $t\in[T]$, the event sequence is given as follows:
At the end of period $T$, a salvage value of $c_nI_{n,T+1} = c_n(I_{n,T} + O_{n,T} - D_{n,T})^+$ is generated for each item $n\in[N]$. For convenience, this term is included in $R_{n,T}$, i.e.,
The initial inventory levels are zero, i.e., $I_{n,1}=0$ for any $n\in[N]$.
According to our definitions of the reward and salvage value, by simple algebra, the total expected reward is given by
From Equation (ref), we found that a myopic policy that optimizes $\mathbb{E}\left[ \sum_{n=1}^N R^+_{n,t}(S_{n,t},D_{n,t}) \right]$ in each period is globally optimal for dynamic inventory problems with capacity constraints under stationary demand conditions (see, e.g., ignall1969optimality; shi2016nonparametric). However, in non-stationary environments where demand distributions evolve over time, obtaining a closed-form global optimal solution over the entire planning horizon is generally intractable. This challenge is further exacerbated when the number of items $N$ and the horizon length $T$ are large, leading to the curse of dimensionality.
Consequently, to ensure tractability and following standard approximation approaches in the literature (e.g., chand2002forecast; levi2007approximation), we adopt a rolling-horizon policy with a single-period lookahead, herein referred to as the Myopic Clairvoyant Policy. Specifically, given the known demand distribution $\{F_{n,t}\}_{n \in [N]}$ and an initial inventory vector $\mathbf{I}_t=(I_{1,t},\dots,I_{N,t})$ in period $t\in[T]$, the myopic optimization problem for that period is formulated as
Although the myopic policy serves as a heuristic in the general non-stationary case, we focus on settings where demand distributions evolve gradually (as formalized later in Assumption (ref)). Under such conditions, the myopic policy effectively captures the system dynamics with minimal loss of optimality compared to the intractable global solution.
In the multi-item setting, the shared capacity constraint prevents base-stock levels from always reaching these unconstrained optimal values, creating interference among items. To solve this constrained optimization problem, we introduce a Lagrangian multiplier $\lambda_t\ge 0$ associated with the capacity constraint (ref).
The solution to this constrained newsvendor problem follows the standard Lagrangian relaxation approach (see, e.g., hadley1963analysis).
While the true demand distributions are unknown in practice, the decision maker must estimate them in order to implement an inventory policy. To facilitate our analysis, we assume that the true demand of item $n$ in period $t$ follows a uniform distribution over the interval $\big[\mu_{n,t}-\alpha_{n,t},\, \mu_{n,t}+\alpha_{n,t}\big]$: \[ D_{n,t} \sim \mathrm{Unif}\big[\mu_{n,t}-\alpha_{n,t},\, \mu_{n,t}+\alpha_{n,t}\big]. \] In practice, the true parameters $(\mu_{n,t}, \alpha_{n,t})$ are unknown to the decision maker. Instead, the policy is based on an estimated demand distribution, \[ \hat{D}_{n,t} \sim \mathrm{Unif}\big[\hat\mu_{n,t}-\hat\alpha_{n,t},\, \hat\mu_{n,t}+\hat\alpha_{n,t}\big]. \] constructed from historical data or auxiliary information.
Consequently, the decision maker applies the myopic single-period solution based on the estimated distribution for period $t$. This yields estimated base-stock levels $\hat{S}_{n,t}$ and an estimated Lagrange multiplier $\hat{\lambda}_t$ that satisfy:
Since inventory control policies depend critically on demand forecasts, we focus on evaluating and comparing different demand forecasting methods.
{We consider two forecasting methods for the demand mean, with the objective to evaluate the effectiveness of the treatment forecast against the control forecast when the resulting ordering decisions are implemented. Specifically, let the binary variable $w \in \{0, 1\}$ denote the forecasting method employed, where $w=0$ represents the control method and $w=1$ represents the treatment method. We model the decision maker's forecast as a random variable following a Normal distribution, i.e., for a given item $n\in[N]$ at time $t\in[T]$, the estimated demand mean is \[ \hat{\mu}_{n,t}(w) \sim \mathcal{N}\big(\tilde{\mu}_{n,t}(w),\, \tilde{\sigma}_{n,t}^2(w)\big), \qquad w\in\{0,1\}, \] where $\tilde{\mu}_{n,t}(w)$ denotes the expected value of the estimator under forecasting method $w$, and $\tilde{\sigma}_{n,t}^2(w)$ represents the estimation variance. } To facilitate analytical tractability in the experimental design and bias analysis, we assume that the decision maker's estimate of the demand range is accurate, i.e., $\hat\alpha_{n,t} = \alpha_{n,t} \ \forall n\in[N],\, t\in[T]$. Therefore, only the mean parameters $\{\hat{\mu}_{n,t}\}_{n\in[N],t\in[T]}$ are subject to misspecification.
The allocation of forecast methods across items and time periods is governed by a treatment assignment matrix \[ \mathbf{W} = (W_{n,t})_{n\in[N],\, t\in[T]} \in \{0,1\}^{N\times T}, \] where $W_{n,t}=1$ indicates that item $n$ uses the treatment forecast in period $t$, and $W_{n,t}=0$ indicates the control forecast.
{ Global Treatment Effect. The primary metric of interest is the Global Treatment Effect (GTE), which measures the difference between the average outcomes when all items are exposed to treatment and when they are all exposed to control, in all time periods. Mathematically, this is defined as \[ \mathrm{GTE} := \frac{1}{NT} \sum_{n=1}^N \sum_{t=1}^T \mathbb{E}\big[ R_{n,t} \mid \mathbf{W} = \mathbf{1}_{N\times T} \big] - \frac{1}{NT} \sum_{n=1}^N \sum_{t=1}^T \mathbb{E}\big[ R_{n,t} \mid \mathbf{W} = \mathbf{0}_{N\times T} \big]. \] This metric captures the total economic impact of fully switching from the control to the treatment method. }
{ Experimental design. The design of experiments (also known as the randomization distribution or simply the design) is a probability distribution over assignment matrices, \[ \mathbb{P}_{\mathbf{W}}: \{0,1\}^{N\times T}\to[0,1], \qquad \sum_{\boldsymbol{w}} \mathbb{P}_{\mathbf{W}}(\boldsymbol{w})=1. \] We impose a marginal constraint $\mathbb{P}(W_{n,t}=1)=p$ for any $n\in[N],t\in[T]$, such that the probability of any specific item-time pair receiving the treatment is fixed at $p$. For each design, we first draw one assignment matrix from the distribution $\mathbb{P}_{\mathbf{W}}$, and then implement this assignment matrix to conduct the experiment. During the experiment, we collect the observed outcomes, and use both the realized assignment matrix and the observed outcomes to estimate the causal estimand. }
IPW estimator. Since it is impossible to simultaneously observe the outcomes under treatment method and control method for the same item and time, we must estimate the GTE using the observed data from the experiment. Given a realization $\boldsymbol{w}$ of the treatment assignment $\mathbf{W}$, we estimate GTE using the the Inverse-Probability-Weighted (IPW) estimator
The bias of the estimator is defined as \[ \text{Bias} := \mathbb{E}\!\left[\widehat{\mathrm{GTE}}\right] - \mathrm{GTE}, \] i.e., the difference between the estimated GTE and the true GTE computed under the corresponding global-treatment and global-control counterfactual policies.
We evaluate the performance of the estimator under three distinct experimental designs: switchback (SW), item-level randomization (IR), and pairwise randomization (PR). These designs differ in how they correlate treatment assignments across items and time periods.
\paragraph*{Switchback (SW).} {The Switchback design synchronizes the treatment assignment across all items within a given time period. In each period $t\in[T]$, with probability $p$, all items are assigned to treatment ($\mathbf{W}_t=\mathbf{1}$), and with probability $1-p$, all items are assigned to control ($\mathbf{W}_t=\mathbf{0}$). Formally, $\{\mathbf{W}_t\}_{t\in[T]}$ are independently and identically distributed (i.i.d.), and $W_{1,t}=\cdots=W_{N,t}\sim\mathrm{Bernoulli}(p)$ for any $t\in[T]$. Figure (ref) provides an example. }
\paragraph*{Item-level Randomization (IR).} In the Item-level Randomization design, the treatment assignment is fixed for each item throughout the entire horizon. Each item is independently assigned to the treatment group for all periods with probability $p$, and to the control group for all periods with probability $1-p$. Therefore, $W_{n,t}$ is constant across $t$ for each item $n$. Figure (ref) provides an example.
\paragraph*{Pairwise Randomization (PR).} To balance the limitations of the previous designs—namely, the strong temporal correlation in switchback experiments and the strong cross-item correlation in item-level randomization—we propose a fully factorized design, which we call pairwise randomization. Here, every entry in the assignment matrix is drawn independently: \[ W_{n,t} \overset{i.i.d.}{\sim} \mathrm{Bernoulli}(p), \qquad \forall n\in[N],t\in[T]. \] Figure (ref) provides an illustration.
In Section (ref), we consider A/B tests that compare inventory policies induced by different demand forecasts. We analyze how these experimental designs distort the estimated global treatment effect and characterize the direction of the resulting bias. In Section (ref), we perform simulated A/B tests based on real-world retail data to empirically validate our theoretical results.
This section examines how improvements in demand forecasting translate into operational gains and how different A/B testing designs recover those gains in multi-item, multi-period inventory systems with a shared capacity constraint. We analyze two stylized regimes.
Taken together, the results show that both the direction and magnitude of experimental bias vary sharply across designs, and that the choice of experimental design interacts critically with whether forecasting improvements affect the mean or the variance of demand estimates.
{The proofs of all results in Section (ref) are provided in Appendix (ref), and the proofs of all results in Section (ref) are provided in Appendix (ref). }
We first study the impact of demand estimation bias on A/B tests. In practice, demand is often underestimated due to censored demand observation, i.e., when inventory is depleted, excess demand is unobserved and lost. Therefore, we want to understand whether a more accurate forecasting method (the treatment) can mitigate this bias by better accounting for censored demand, and whether A/B tests under different experimental designs can accurately estimate the corresponding treatment effect.
We focus on a scenario where both the control and treatment methods underestimate the true mean demand, but the treatment exhibits reduced bias. To facilitate the analysis, we assume zero estimation variance ($\tilde\sigma_{n,t}(w)=0$, or equivalently $\hat \mu_{n,t}(w) = \tilde \mu_{n,t}(w)$) for all $w\in\{0,1\},n\in[N],t\in[T]$ in this subsection, and defer the analysis of estimation variance to Section (ref). More specifically, we consider the following scenario.
Scenario 1: For each item $n \in [N]$, period $t \in [T]$, there exists a forecast error parameter $\Delta$ such that the estimated demand mean is given by: \[ \hat \mu_{n,t}^{(\Delta)} = \mu_{n,t} + \Delta \alpha_{n,t}, \] where $\Delta$ perturbs the predicted mean proportionally to the demand uncertainty parameter $\alpha_{n,t}$. We consider the case where the treatment and control satisfy \[ \hat \mu_{n,t} (0) = \hat \mu_{n,t}^{(\Delta_0)},\ \hat \mu_{n,t} (1) = \hat \mu_{n,t}^{(\Delta_1)}, \ \Delta_0 < \Delta_1 \le 0. \]
We make the following assumptions in our analysis.
Assumption (ref) precludes the overshooting of order-up-to targets between consecutive periods. Under this assumption, $\hat S_{n,t}$ equals the newsvendor quantile estimation of the demand, i.e. $\hat F_{n,t}^{-1}\Big( (\tfrac{b_n - c_n - \hat \lambda_t}{b_n -c_n+ h_n})^+ \Big)$. Essentially, it ensures that the current decision $\hat S_{n,t}$ does not depend on the past treatment/control decisions $\mathbf{W}_{N\times (t-1)}$, thereby isolating the direct effect of the current forecast. To emphasize the dependence on $\mathbf{W}_t$, we denote the base-stock levels and the myopic Lagrange multiplier by $\hat{S}_{n,t}(\mathbf{W}_{t})$ and $\hat{\lambda}_t(\mathbf{W}_t)$ in the remaining part of the paper. {In addition, when all items have the same forecast error parameter $\Delta$ at time $t$ in the SW experiment, we simply use the notations $\hat{S}_{n,t}^{(\Delta)}$ and $\hat \lambda_t^{(\Delta)}$. }We note that, however, the past decisions still affect the current profit $R_{n,t}$ via the initial inventory level $I_{n,t}$.
In the uniform demand setting, this assumption holds naturally when demand does not exhibit abrupt fluctuations across adjacent periods, and when the lower support of the demand distribution, $\underline D_{n,t}=\mu_{n,t}-\alpha_{n,t}$, is sufficiently large relative to the magnitude of forecast-induced changes in the target base-stock levels, which are of order $O(\alpha_{n,t})$.
{ Denote \[ u_n = \frac{b_n-c_n}{b_n -c_n+ h_n},\qquad v_n = \frac{1}{b_n -c_n+ h_n},\qquad \zeta_t=\frac{\sum_{n=1}^N \big[\mu_{n,t} + (2u_n - 1)\alpha_{n,t}\big] - B}{2\sum_{n=1}^N \alpha_{n,t}v_n}. \] We further make the following assumption:
}
{ Assumption (ref) implies that the Lagrange multiplier $\lambda_t^*$—which measures the marginal value of relaxing the capacity constraint—remains below the unit margins $b_n-c_n$ for all $n$, so that all base-stock levels lie strictly in the interior of the demand support. The assumption is mild when capacity is not extremely scarce relative to baseline demand and unit margins are not too small. In the homogeneous-item case, where all items share the same parameters $(b,c,h,\mu_t,\alpha_t)$, the assumption is equivalent to \[ B > N(\mu_t-\alpha_t), \] i.e., the total capacity exceeds the aggregate lower support of demand. This corresponds to a moderately constrained regime in which capacity may bind, but not so severely that any item’s order-up-to level collapses to the lower support of demand. }
Essentially, Assumption (ref) together with Assumption (ref) guarantees that the quantile term in the myopic optimal base-stock level (ref) lies strictly within the interval $(0,1)$, i.e., for any $n\in[N]$ and $t\in[T]$,
Under our uniform-demand assumption, this implies that the base-stock levels are affine in the Lagrange multiplier. {To see this, note that at $\Delta=0$, the KKT conditions determine a multiplier $\lambda_t^*\ge0$ for each period $t$. From (ref) with $\hat{\mu}_{n,t}=\mu_{n,t}$ and $\hat{\alpha}_{n,t}=\alpha_{n,t}$, we see that under Assumption(ref) this multiplier admits an explicit representation
which further implies that condition (ref) holds true for any $n\in[N]$ and $t\in[T]$. }
{ Depending on the value of $\lambda_t^*$, we can further categorize the whole system in period $t$ into two cases:
We also note that the interior case ensures that $S_{n,t}^*>\mu_{n,t}-\alpha_{n,t}$ for all $n$, meaning every item's order-up-to level lies strictly within the support of its demand distribution. } { Although the strictly interior case is implied by Assumption (ref) only at $\Delta=0$, the next lemma show that, due to the continuity of the KKT system, this condition should remain valid over an interval $[\underline{\Delta},0]$ in the SW experiment where all items have the same forecast error parameter $\Delta$. Within this interval, the solution never enters a partially binding regime where $\hat \lambda_t$ is positive but some order-up-to level hits the lower support of demand. That is, when all items share a forecast error parameter $\Delta\in[\underline{\Delta},0]$, we have $\hat S_{n,t}>\mu_{n,t}-\alpha_{n,t}$ whenever $\hat\lambda_t>0$. This facilitates the analysis of $GTE$ and the bias in the SW experiment. } {
}
Our first result characterizes the monotonic effect of reducing the forecast bias on expected profits. In particular, it shows that, when the forecast error parameter lies within the interval $[\underline{\Delta},0]$, better demand estimates always lead to higher expected profits.
The proposition formalizes the intuition that reducing downward forecast bias improves operational performance: when the treatment reduces the common bias parameter $\Delta$ (i.e., $\Delta_1>\Delta_0$ with both close to $0$), the global treatment effect is nonnegative.
We now turn to the performances of the IPW estimator under various experimental designs. A fundamental challenge in dynamic inventory experiments is that treatment assignment affects not only current-period decisions but also the evolution of inventories, thereby inducing serial correlation in outcomes. The switchback experimental design suffers from temporal correlation because it assigns all items to the same group within a period. The following theorem quantifies the resulting bias and shows that SW systematically underestimates the global treatment effect.
The bias expression in Theorem (ref) shows that the sign of ${Bias}^{SW}$ is entirely driven by the difference in expected leftover inventory between the treatment and control regimes. When the treatment improves forecast accuracy (i.e., $\Delta_1>\Delta_0$ with both in $[\underline{\Delta},0]$), the myopic base-stock rule produces higher order-up-to levels and therefore larger expected end-of-period inventory under treatment.
Under the switchback design, however, leftover inventory generated in a treated period might become the initial inventory of the subsequent control period. In contrast, under the full-treatment counterfactual, this inventory would have been carried forward into another treated period. This reallocates leftover inventory from the treated potential-outcome path to the control path, effectively advantaging the control group. Since the naive IPW estimator cannot reconstruct the correct counterfactual inventory-carryover sequence, it systematically understates the difference in profits between treatment and control, yielding a nonpositive bias.
In contrast to SW, the item-level randomized (IR) design fixes each item in either treatment or control for the entire horizon, thereby eliminating temporal variation but inducing persistent cross-sectional heterogeneity. The next theorem shows that, under the same demand-bias scenario, IR produces a {nonnegative} bias: the IPW estimator systematically {overestimates} the global treatment effect.
The nonnegative bias of IR is driven by the capacity constraint. When treated and control items coexist in the same period, the higher forecast means of the treated items push their target base-stock levels upward. Because capacity is fixed, this expansion necessarily crowds out the control items, lowering their base-stock levels even further than in the global-control counterfactual. Symmetrically, treated items receive more inventory than they would under the global-treatment policy because they face weaker competition from the underestimated control items. Both distortions move the observed treated–control contrast away from the true GTE in the same direction, yielding a systematically upward-biased estimator.
The previous results show that SW and IR sit on opposite sides of the truth: SW delivers a downward-biased estimate of GTE, whereas IR yields an upward-biased one. Pairwise Randomization (PR) blends the two dependence structures that drive the biases in SW and IR. Like SW, it generates temporal dependence in inventory carryover, inducing a downward force on the estimator; like IR, it creates cross-sectional competition under a binding capacity constraint, inducing an upward force. Because these forces operate in opposite directions, the resulting bias is upper-bounded by the bias under IR.
The bias decomposition in Theorem (ref) contains two parts: a pair of IR-type terms that push the estimator upward, and an inventory-carryover term that pushes it downward. The latter term arises because PR allows treated and control items to alternate across periods, partially offsetting the overstatement of treatment benefits that occurs under IR. The inequality $Bias^{PR} \le Bias^{IR}$ shows that pairwise randomization never worsens the bias relative to IR and typically yields an estimate closer to zero.
{We now turn to the impact of demand estimation variance on A/B tests. Specifically, we consider the setting where treatment and control groups estimate the same demand mean, but the treatment group enjoys strictly smaller forecast variance. This scenario is particularly relevant for evaluating modern machine learning upgrades, which often reduce uncertainty (narrowing confidence intervals) even when point estimates remain unchanged. }
Scenario 2: For each item $n\in[N]$, period $t\in[T]$, and group $w\in\{0,1\}$, \[ \hat\mu_{n,t}(w)\sim\mathcal N(\mu_{n,t},\tilde\sigma_{n,t}^2(w)), \qquad \tilde\sigma_{n,t}^2(1)<\tilde\sigma_{n,t}^2(0). \] Define the estimation error $\epsilon_{n,t}(w)=\hat\mu_{n,t}(w)-\mu_{n,t}$.
Modern large-scale inventory systems often involve thousands or millions of items sharing a joint warehouse capacity. This motivates a mean–field regime where $N\to\infty$, {the warehouse capacity $B^{(N)}\rightarrow\infty$ (we add a superscript notation $^{(N)}$ to the corresponding terms in the $N$-th system),} and cross–sectional averages converge to deterministic limits. In the $N$-th system, the Lagrange multiplier under the assignment $\mathbf{W}_{t} $ is defined as the KKT multiplier \[ \hat\lambda_{t}^{(N)}(\mathbf W_{t}) := \max\Bigl\{0,\ \inf\{\lambda\ge 0:H_N(\lambda)\le 0\}\Bigr\}, \] where \[ H_N(\lambda) =\frac{1}{N}\sum_{n=1}^N \hat S_{n,t}^{(N)}(\lambda) -\frac{B^{(N)}}{N}, \qquad \hat S_{n,t}^{(N)}(\lambda) = \max \{ I_{n,t},\hat\mu_{n,t}(W_{n,t})+\alpha_{n,t}\bigl(2(u_n-v_n\lambda)^+ - 1\bigr)\}, \] and \[ u_n=\frac{b_n-c_n}{b_n-c_n+h_n},\qquad v_n=\frac{1}{b_n-c_n+h_n}. \]
To ensure the validity of the mean-field limit, we further impose the following assumptions.
Item-level independence ensures that cross-sectional averages converge by the law of large numbers, which is essential for the mean-field limit. {The independence between forecasts and demands ensures that forecast uncertainty impacts profits solely through the stocking decision (the base-stock level) rather than through spurious correlations, which allows us to use convex-order comparisons in the proofs for GTE and bias. }
Uniform moment bounds rule out extreme items with unbounded demand, costs, or forecast variance, and guarantee the integrability of revenues and leftover inventories in the proofs. The lower bound on $b_n+h_n-c_n$ ensures that the newsvendor quantile is well-defined and prevents degeneracy in the base-stock policy.
Assumptions (ref) and (ref) jointly ensure that the normalized capacity slack function admits a deterministic mean-field limit. {Specifically, the first assumption controls the linear components of the aggregate base-stock level, while the second ensures that the nonlinearity introduced by the truncation (the $(\cdot)^+$ operator) smooths out across the items. } Together, they ensure that the KKT condition for the capacity constraint converges to a deterministic fixed-point equation in the limit $N\to\infty$.
The following lemma shows that, in the mean–field limit, the random KKT multiplier becomes deterministic.
{In the mean-field limit, randomness from both treatment assignments and noisy mean forecasts is averaged out at the system level. {The capacity constraint is enforced by a deterministic multiplier $\lambda_t^*$ common to all items.} The only remaining difference between treatment and control groups is the idiosyncratic variance of the forecast errors $\epsilon_{n,t}(w)$.}
We now present the main results on the GTE and the asymptotic bias of different experimental designs.
In Scenario 2, the treatment reduces the variance of the mean forecast while keeping the mean unchanged. {Because $s\mapsto(s-D)^+$ is convex and the profit function penalizes the overstock $\mathbb{E}(s-D)^+$, a reduction in forecast variance decreases expected overstock penalties.} Consequently, smaller forecast variance leads to lower expected holding cost without affecting expected sales, resulting in a weakly higher expected profit for the treatment group in the mean-field limit.
In contrast to Scenario 1, where SW underestimates the GTE, here the switchback design {overestimates} the value of the treatment in the large-$N$ regime. { The mechanism is a “starvation" spillover. Intuitively, under treatment, the more precise forecasts reduce the variability of base-stock levels and lead to lower expected leftovers. When the system switches back to control, these depleted leftovers force the control group to place larger orders than under the global-control counterfactual to restore inventory levels. } As a result, the cost difference between treatment and control is overstated, and the naive IPW estimator {overestimates} the true treatment effect, yielding a positive bias.
In item-level randomization, each item is permanently assigned to either treatment or control, so there is no temporal switching at the item level. In the mean-field limit, the capacity multiplier $\lambda_t^*$ is shared by both groups and becomes deterministic. Therefore, the environment faced by the treatment (/control) group in IR coincides asymptotically with that in the corresponding global-treatment (/global-control) regime. Thus, the naive IPW estimator compares two groups that differ only in forecast variance, without additional spillover distortions, and the IR design becomes asymptotically unbiased.
Pairwise randomized experiments (PR) randomize at both the item and time dimensions. In the mean-field limit, cross-sectional randomization does not create a persistent imbalance between treatment and control groups because both share the same deterministic capacity Lagrange multiplier $\lambda^*_t$ in each period $t$. The remaining bias arises from the same intertemporal leftover spillover as in switchback designs: lower-variance forecasts under treatment reduce leftovers in treated periods, which in turn distort initial inventories in subsequent periods. As a consequence, PR inherits the same asymptotic positive bias as SW, while IR remains unbiased.
In the variance-driven regime of Scenario 2, the SW and PR designs share the same asymptotic bias, which is nonnegative, whereas IR is asymptotically unbiased. PR inherits both the temporal dependence structure of SW and the cross-sectional heterogeneity of IR. Here the temporal component dominates, so that the PR estimator converges to the same upward-biased limit as SW and stays above the IR estimator for large $N$. This stands in sharp contrast to Scenario 1 and highlights that the direction of bias is sensitive to whether the treatment improves the {mean} or the {variance} of forecasts.
We summarize our main theoretical results in Table (ref).
We conduct A/B tests using FreshRetailNet-50K wang2025freshretailnet, a large-scale fresh retail dataset from Dingdong Fresh (China). It contains 50{,}000 store--product time series of hourly sales from 898 stores in 18 major cities (March--June 2024), along with verified stockout annotations. We use the first 90 days to train demand predictors and evaluate inventory outcomes on the final 7-day horizon.
Our theory is formulated for demand distributions and their implied mean forecasts. To evaluate our findings without imposing a specific parametric demand distribution, we use a trace-driven simulation based on recovered historical demand. We apply a demand-imputation model in wang2025freshretailnet to recover latent (uncensored) demand and use the recovered series $d_{n,t}$ in policy simulation and A/B evaluation. This reduces the mechanically induced correlation between demand level and stockout rate and helps separate forecasting effects from censoring artifacts. Accordingly, each forecaster outputs a point prediction $\hat d_{n,t}$ (interpreted as an estimate of the demand mean) rather than a full predictive distribution.
Policy implementation (myopic greedy allocation). Because the simulation uses point forecasts, we implement the myopic policy as a simple greedy allocation rule. In each period $t$, the policy ranks items by unit margin $b_n - c_n$ in descending order and allocates inventory to satisfy the predicted demand $\hat d_{n,t}$ in that order until the shared capacity is reached. This implementation preserves the key mechanism in our theory: under a binding capacity constraint, higher-margin items can crowd out lower-margin items.
Forecast Scenarios. To match the regimes studied in our theory, we construct treatment and control forecast series with targeted empirical error patterns. We measure forecast error using Weighted Absolute Percentage Error (WAPE) and Weighted Percentage Error (WPE):
Scenario 1 (reduction of downward bias). We create this regime by selecting control forecasts that substantially underestimate demand (large negative WPE), which mimics the bias that can arise when models are trained on stockout-censored sales. We select treatment forecasts with much smaller bias, so that $\mathrm{WPE}_{\text{control}} < \mathrm{WPE}_{\text{treatment}} \approx 0$. Table (ref) reports the resulting treatment--control forecast gaps.
Scenario 2 (reduction of estimation variance). We create this regime by selecting treatment forecasts with lower error magnitude (smaller WAPE) than the control, while keeping both forecasts approximately unbiased (WPE near zero). This isolates the effect of reduced forecast noise. Table (ref) summarizes the selected pairs.
Price Parameters. Because real transactional prices and costs are unavailable in FreshRetailNet-50K, we synthesize product-level price and cost parameters. For SKU $i$ in store $s$ on day $t$, we generate \[ \text{price}_{i,s,t} = \text{base}_i \cdot \text{category}_i \cdot \text{store}_s \cdot \text{holiday}_t \cdot \text{stock}_{i,s,t}, \] where $\text{base}_i$ and the multiplicative factors are sampled from narrow uniform ranges (with $\text{base}_i$ centered around 50 and other multipliers centered around 1.0). We then set ordering and holding costs as fixed fractions of $\text{price}_{i,s,t}$: $\text{ordering cost}_{i,s,t} \sim \mathcal{U}[0.3, 0.6] \times \text{price}_{i,s,t}$ and $\text{holding cost}_{i,s,t} \sim \mathcal{U}[0, 0.3] \times \text{ordering cost}_{i,s,t}$. The realized sale price is $\text{price}_{i,s,t}\times \text{discount}_{i,s,t}$.
We compare the following demand forecasters:
Stockout-substitution mechanism. In practice, when a product is sold out, customers may purchase substitute products instead. To model this behavior, we optionally apply a stockout-substitution mechanism: when an item stocks out, its unmet demand is redistributed to alternative SKUs within the same store rather than being entirely lost. We draw the redistribution from a multinomial distribution parameterized by a precomputed substitution matrix based on product-hierarchy similarity (closer substitutes receive higher probability).
We report the substitution ratio as \[ \text{sub-ratio}=\frac{\text{demand received via substitution}}{\text{total demand}}, \] averaged across all SKUs. In our experiments, the sub-ratio ranges from 19% to 33% in Scenario 1 and from 5% to 20% in Scenario 2.
We evaluate three experimental designs: switchback (SW), item-level randomization (IR), and pairwise randomization (PR). We use a static capacity $B_s$ for each store $s$. We calculate the median historical daily sales across all products, denoted as $\bar{d}_{\text{median}}$. The store capacity is set as $B_s = \rho \cdot N_s \cdot \bar{d}_{\text{median}}$, where $N_s$ is the number of SKUs in store $s$ and $\rho$ is the capacity factor varying from $0.6$ to $1.8$. For each design and each capacity level, we run 30 independent randomizations and report the empirical distribution of the estimated GTE (violin plots).
In Scenario 1, treatment and control forecasts differ primarily in systematic bias, which we construct using stockout-induced censoring. Specifically, the control forecaster is trained on the original (censored) sales series and its predictions are evaluated against the recovered demand on the test horizon, yielding systematic underestimation (negative WPE). In contrast, the treatment forecaster is trained on the recovered demand and produces a substantially less biased forecast ({WPE} $\le 0.10$). Table (ref) summarizes the out-of-sample forecast gaps for the treatment--control pairs.
Figures (ref)--(ref) are consistent with the theoretical predictions for Scenario 1. First, SW exhibits negative bias and substantially higher dispersion than IR and PR. A likely reason is limited temporal replication: SW assigns treatment at the level of time blocks within a 7-day horizon, whereas IR and PR effectively average over a much larger cross-sectional sample each day (approximately 50{,}000 SKUs), leading to lower variance. Second, IR tends to be positively biased (overestimating the treatment effect), consistent with cross-sectional interactions created by the shared capacity constraint. Third, PR typically reduces the positive bias observed under IR; however, PR may remain negatively biased when intertemporal carryover effects are non-negligible. We further note that, in this numerical study, the no-overshoot assumption (Assumption (ref)) may not always hold, which further underscores the robustness of our insights.
The capacity regime affects the bias patterns in a way that is informative about the underlying mechanisms. As the capacity becomes tighter, the bias of SW moves closer to zero. When the capacity constraint is binding, replenishment decisions in treatment and control periods become more similar, which limits the extent to which inventory carryover can distort comparisons across switchback blocks. Conversely, as capacity becomes looser, the bias of IR approaches zero. In this regime, the capacity constraint is rarely binding, so cross-sectional competition for budget weakens and the “crowding” channel that drives IR bias is attenuated. At intermediate capacity levels, PR often balances these two channels and yields smaller overall bias than SW or IR.
When stockout substitution is activated, GTEs tend to increase because a portion of lost sales is recovered through substitution. Nevertheless, the qualitative pattern across experimental designs remains similar: SW tends to be negatively biased, IR tends to be positively biased, and PR typically lies between SW and IR (often closer to zero than IR).
In Scenario 2, we isolate the effect of forecast noise: treatment and control forecasts have similar mean bias (WPE near zero), but the treatment has smaller error magnitude (lower WAPE). We construct treatment--control pairs so that both groups are approximately unbiased while the treatment has strictly smaller WAPE (Table (ref)).
Figures (ref)--(ref) show positive treatment effects across all tested model pairs, consistent with the theoretical prediction that reducing estimation variance improves expected profit. In these simulations, IR remains close to unbiased with small dispersion, whereas SW and PR exhibit positive bias.
Compared with Scenario 1, the capacity regime has a weaker impact in Scenario 2. A plausible explanation is that Scenario 2 mainly reduces the dispersion of $\hat\mu$ around an approximately unbiased mean; once the mean-field multiplier stabilizes (Lemma (ref)), tightening or loosening capacity has a smaller marginal effect on the relative performance of two nearly unbiased forecasts in a 7-day experiment. Consistent with this explanation, the violin plots are qualitatively similar across the tight/medium/loose regimes.
With substitution enabled, the treatment effect remains positive and the qualitative comparison among SW, IR, and PR is unchanged.
In this paper, we study A/B testing strategies in multi-item, multi-period inventory systems with lost sales and capacity constraints. When the demand estimators used by the treatment and control primarily differ in their means, we show that switchback designs systematically underestimate, whereas item-level randomization systematically overestimates, the global treatment effect. When the demand estimators primarily differ in their variances, switchback designs typically overestimate, while item-level randomization is asymptotically unbiased. Finally, we show that pairwise randomization can serve as a superior alternative in certain regimes. Based on our theoretical and numerical findings, we offer the following practical recommendations: