Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
97,493 characters · 28 sections · 58 citation commands
SHOPPER: A Probabilistic Model of Consumer Choice with Substitutes and Complements
Large-scale shopping cart data provides unprecedented opportunities for researchers to understand consumer behavior and to predict how it responds to interventions such as promotions and price changes. Consider the shopping cart in (ref). This customer has purchased items for their baby (diapers, formula), their dog (dog food, dog biscuits), some seasonal fruit (cherries, plums), and the ingredients for tacos (taco shells, salsa, and beans). Shopping cart datasets may contain thousands or millions of customers like this one, each engaging in dozens or hundreds of shopping trips.
In principle, shopping cart datasets could help reveal important economic quantities about the marketplace. They could help evaluate counterfactual policies, such as how changing a price of a product will affect the demand for it and for other related products; and they could help characterize consumer heterogeneity, which would allow firms to consider different interventions for different segments of their customers. However, large-scale shopping cart datasets are too complex and heterogenous for classical methods of analysis, which necessarily prune the data to a handful of categories and a small set of customers. In this paper, our goal is to develop a method that can usefully analyze large collections of complete shopping baskets.
Shopping baskets are complex because, as the example shopper in (ref) demonstrates, many interrelated forces are at play when determining what a customer decides to buy. For example, the customer might consider how well the items go together, their own personal preferences and needs (and whims), the purpose of the shopping trip, the season of the year, and, of course, the prices and the customer's personal sensitivity to them. Moreover, these driving forces of consumer behavior are unobserved elements of the market. Our goal is to extract them from observed datasets of customers' final purchases.
To this end we develop \oldtextsc{\scalefont{1.10}shopper}, a sequential probabilistic model of market baskets. \oldtextsc{\scalefont{1.10}shopper}\ uses interpretable components to model the forces that drive customer choice, and we designed \oldtextsc{\scalefont{1.10}shopper}\ to capture properties of interest regarding how items interact with other items; in particular, we are interested in answering counterfactual queries with respect to item prices. We also develop an efficient posterior inference algorithm to estimate these forces from large-scale data. We demonstrate \oldtextsc{\scalefont{1.10}shopper}\ by analyzing data from a major chain grocery store Che2012. We found that \oldtextsc{\scalefont{1.10}shopper}\ provides accurate predictions even under price interventions, and that it also helps identifying complementary and substitutable pairs of items.
\oldtextsc{\scalefont{1.10}shopper}\ is a hierarchical latent variable model of market baskets for which the generative process comes from an assumed model of consumer behavior. In the language of social science, \oldtextsc{\scalefont{1.10}shopper}\ is a structural model of consumer behavior, where the elements of the model include the specification of consumer preferences, information, and behavior (i.e., utility maximization). In the language of probabilistic models, it can equivalently be defined by a generative model.
\oldtextsc{\scalefont{1.10}shopper}\ posits that a customer walks into the store and chooses items sequentially. Further, the customer might decide to stop shopping and pay for the items; this is the “checkout” item. At each step, the customer chooses among the previously unselected items, while conditioning on the items already in the basket. The customer's choices also depend on various other aspects: the prices of the items and the customer's sensitivities to them, the season of the year, and the customer's general shopping preferences {(which are specific for each customer)}.
One key feature of \oldtextsc{\scalefont{1.10}shopper}\ is that each possible item is associated with latent attributes {$\alpha_c$}, vector representations that are learned from the data. This is similar in spirit to methods from machine learning that estimate semantic attributes of vocabulary words by analyzing the words close to them in sentences Bengio2003. In \oldtextsc{\scalefont{1.10}shopper}\, “words” are items and “sentences” are baskets of purchased items.
\oldtextsc{\scalefont{1.10}shopper}\ uses the latent attributes in two ways. First, they help represent the existing basket when considering which item to select next. Specifically, each possible item is also associated with a vector of interaction coefficients {$\rho_c$}, parameters that represent which kinds of basket-level attributes it tends to appear with. For example, the interaction coefficients and attributes can capture that when taco shells are in the basket, the customer has a high probability of choosing taco seasoning. Second, they are used as the basis for representing customer preferences. Each customer in the population also has a vector of interaction coefficients {$\theta_u$}, which we call “preferences,” that represent the types of items that they tend to purchase. For example, the customer preferences and attributes can capture that some customers tend to purchase baby items or dog food.
{Mathematically, at the $i$th step of the sequential process, the customer chooses item $c$ with probability that depends on the latent features of item $c$, on her preferences $\theta_u$, and on the representation of the items that are already in the basket, $\sum_{j=1}^{i-1} \alpha_{y_j}$ {(where $y_j$ indicates the item purchased at step $j$)}. In its vanilla form, }\oldtextsc{\scalefont{1.10}shopper}{ posits that this probability takes a log-bilinear form,}
(ref){ provides more details about the model and also describes how to incorporate price and seasonal effects.}
When learned from data, the latent attributes {$\alpha_c$} capture meaningful dimensions of the data. For example, (ref) illustrates a two-dimensional projection of learned latent item attributes. Similar items are close together in the attribute space, even though explicit attributes (such as category or purpose) are not provided to the algorithm.
In the simplest \oldtextsc{\scalefont{1.10}shopper}\ model, the customer is myopic: at each stage they do not consider that they may later add additional items into the basket. This assumption is problematic when items have strong interaction effects, since it is likely that a customer will consider other items that complement the currently chosen item. For example, if a customer considers purchasing taco seasoning, they should also consider that they will later want to purchase taco shells.
To relax this assumption, we include a second key feature of \oldtextsc{\scalefont{1.10}shopper}, called “thinking ahead,” where the customer considers the next choice in making the current choice. For example, consider a scenario where taco shells have an unusually high price and where the customer is currently contemplating putting taco seasoning in the basket. As a consequence of thinking ahead, the model dampens the probability of choosing seasoning because of the high price of its likely complement, taco shells. \oldtextsc{\scalefont{1.10}shopper}\ models one step of thinking ahead. (It may be plausible in some settings that customers think further ahead when shopping; we leave this extension for future work.)
We fit \oldtextsc{\scalefont{1.10}shopper}\ to a large data set of market baskets, estimating the latent features of the items, seasonal effects, preferences of the customers, and price sensitivities. We evaluate the approach with held-out data, used in two ways. First, we hold out items from the data at random and assess their probability with the posterior predictive distribution. This checks how well \oldtextsc{\scalefont{1.10}shopper}\ (and our corresponding inference algorithm) captures the distribution of the data. Second, we evaluate on held-out baskets in which items have larger price variation compared to their average values. This evaluation assesses how well \oldtextsc{\scalefont{1.10}shopper}\ can evaluate counterfactual price changes. With both modes of evaluation, we found \oldtextsc{\scalefont{1.10}shopper}\ gave better predictions than state-of-the-art econometric choice models and machine learning models for item consumption{, which are based on either user/item factorization, such as factor analysis or Poisson factorization }Gopalan2015{, or on item/item factorization, such as exponential family embeddings }Rudolph2016.
Further, \oldtextsc{\scalefont{1.10}shopper}\ can help identify pairs of items that are substitutes and complements---where purchasing one item makes the second item less or more attractive, respectively. This is one of the fundamental challenges of analyzing consumer behavior data. {Formally, two items are complements if the demand of one is increased when the price of the other decreases; they are substitutes if the demand for one rises when the price of the other item increases. Studying complementary and substitutable items is key to make predictions about the (joint) demand.}
Notice that being complements is more than the propensity for two items to be co-purchased; items may be purchased together for many reasons. As one reason, two items may be co-purchased because people who like one item tend to also like the other item. For example, baby formula and diapers are often co-purchased but they are not complements---when baby formula is unavailable at the store, it does not affect the customer's preference for diapers.
\oldtextsc{\scalefont{1.10}shopper}\ can disentangle complementarity from other sources of co-purchase because prices change frequently in the dataset. If two items are co-purchased but not complements, increasing the price of one does not decrease the purchase rate of the other. Using the item-to-item interaction term in \oldtextsc{\scalefont{1.10}shopper}, we develop a measure to quantify how complementary two items are. (In this sense, \oldtextsc{\scalefont{1.10}shopper}\ corroborates the theory of athey1998empirical among others, who show that if there is sufficient variation in the price of each item then it is theoretically possible to separate correlated preferences from complementarity.)
Now we turn to substitutes, where purchasing one item decreases the utility of another, e.g., two brands of (otherwise similar) taco shells. Although in principle substitutes can be treated symmetrically to complements (increasing the price of one brand of taco shells increases the probability that the other is purchased if they are substitutes), in practice the two concepts are not equally easy to discover in the data. The difference is that in most shopping data, purchase probabilities are very low for all items, and so most pairs of items are rarely purchased together. In this paper, we introduce an alternative way to find relationships among products, “exchangability.” Two products are exchangeable if they tend to have similar pairwise interactions with other products; for example, two brands of taco shells would have similar interactions with products such as tomatoes and beans. Of course, products that are usually purchased together, like hot dogs and buns, will also have similar pairwise interactions with other products (such as ketchup). Our results suggest that items that are exchangeable and not complementary tend to be substitutes (in the sense of being in the same general product category in the grocery store hierarchy).
We develop \oldtextsc{\scalefont{1.10}shopper}, a sequential probabilistic model of market baskets. \oldtextsc{\scalefont{1.10}shopper}\ treats each choice as one over the set of available items, where the attributes of each item are latent variables.
We describe \oldtextsc{\scalefont{1.10}shopper}\ in three stages. (ref) describes a basic model of sequential choice with latent item attributes; (ref) extends the model to capture user heterogeneity, seasonal effects, and price; (ref) develops “thinking ahead,” where we model each choice in a way that considers the next choice as well.
\oldtextsc{\scalefont{1.10}shopper}\ comes with significant computational challenges, both because of its complex functional form and the size of the data that we would like to analyze. We defer these challenges to (ref), where we develop an efficient variational algorithm for approximate posterior inference.
We describe the basic model in terms of its generative process or, in the language of economics and marketing, in terms of its structural model of consumer behavior. Each customer walks into the store and picks up a basket. She chooses items to put in the basket, one by one, each one conditional on the previous items in the basket. The process stops when she purchases the “checkout” item, which we treat as a special item that ends the trip.
From the utility-maximizing perspective, \oldtextsc{\scalefont{1.10}shopper}\ works as follows. First, the customer walks into the store and obtains utilities for each item. She considers all of the items in the store and places the highest-utility item in her basket. In the second step, once the first item is in the basket, the customer again considers all remaining items in the store and selects the highest-utility choice. However, relative to the first decision, the utilities of the products change. First, the specification of the utility allows for interaction effects---items may be substitutes (e.g., two brands of taco seasoning) or complements (e.g., taco shells and taco seasoning), in the sense that the utility of an item may be higher or lower as a result of having the first item in the basket. Second, the customer's utilities have a random component that changes as she shops. This represents, for example, changing ideas about what is needed, or different impressions of products when reconsidering them. (Another extension of the model would be to allow for correlation of the random component over choice events within a shopping trip; we leave this for future work.) The customer repeats this process---adjusting utilities and choosing the highest-utility item among what is left in the store---until the checkout item is the highest-utility item.
{In some applications, it may not be reasonable to model the consumer as considering all items. In a supermarket, a customer might consider one part of the store at the time; in an online store, a customer may only consider the products that appear in a search results page. Corresponding extensions to }\oldtextsc{\scalefont{1.10}shopper}{ are straightforward but, for simplicity, we do not include them here. Modeling customers as choosing the most desirable items in the store first ensures that, among goods that are close substitutes, they select the most desirable one.}
We describe {the sequential choice model} more formally. Consider the $t$th trip and let $n_t$ be the number of choices, i.e., the number of purchased items. Denote the (ordered) basket $\mathbf{y}_t = (y_{t1}, \ldots, y_{tn_t})$, where each $y_{ti}$ is one of $C$ items and choice $y_{tn_t}$ is always the checkout item. Let $\mathbf{y}_{t,i-1}$ denote the items that are in the basket up to position $i$, $\mathbf{y}_{t,i-1} = (y_{t1}, \ldots, y_{t,i-1})$.
Consider the choice of the $i$th item. The customer makes this choice by selecting the item $c$ that has maximal utility $U_{t,c}(\mathbf{y}_{t,i-1})$, which is a function of the items in the basket thus far. The utility of item $c$ is
Here $\Psi(c, \mathbf{y}_{t, i-1})$ is the deterministic part of the utility, a function of the other items in the basket. We define $\Psi(c,\mathbf{y}_{t,i-1}) \equiv -\infty$ for $c \in \mathbf{y}_{t,i-1}$, so that the choice is effectively over the set of non-purchased items. The random variable $\epsilon_{t,c}$ is assumed to follow a zero-mean Gumbel distribution (generalized extreme value type-I), which is independent across items.
This behavioral rule---and particularly the choice of Gumbel error terms---implies that the conditional choice probability of an item $c$ that is not yet in the basket is a softmax,
Notice the denominator is a sum over a potentially large number of items. For example, (ref) analyzes data with nearly six thousand items. (ref) describes fast methods for handling this computational bottleneck.
(ref) gives a model of observed shopping trips, from which we can infer the utility function. The core of \oldtextsc{\scalefont{1.10}shopper}\ is in the unnormalized log probabilities $\Psi(c, \mathbf{y}_{t,i-1})$, which correspond to the mean utilities of the items. We assume that they have a log-linear form,
There are two terms. The first term $\psi_{tc}$ is a latent variable that varies by item and by trip; below we use this term to capture properties such as user heterogeneity, seasonality, and price. We focus now on the second term, which introduces two important latent variables: the per-item interaction coefficients $\rho_c$ and the item attributes $\alpha_c$, both real-valued $K$-vectors. When $\rho_c^\top \alpha_{c'}$ is large, then having $c'$ in the basket increases the benefit to the customer of placing $c$ in the basket (the items are complements in the customer's utility); conversely, when the expression is negative, the items are substitutes.
Unlike traditional factorization methods, the factorization is asymmetric. We interpret $\alpha_c$ as latent item characteristics or attributes; we interpret $\rho_c$ as the interaction of item $c$ with other items in the basket, as described by their attributes (i.e., their $\alpha$'s). These interactions allow that even though two items might be different in their latent attributes (e.g., taco shells and beans), they still may be co-purchased because they are complements to the consumer. It can also capture that similar items (e.g., two different brands of the same type of taco seasoning) may explicitly not be purchased together---these are substitutable items. Below we also allow that the latent attributes $\alpha_c$ further affect the item's latent mean utility on the trip $\psi_{tc}$.
We also note that the second term has a scaling factor, $1/(i-1)$. This scaling factor captures the idea that in a large basket, each individual item has a proportionally smaller interaction with new purchases. This may be a more reasonable assumption in some scenarios than others, exploring alternative ways to account for basket size is a direction for future work.
Finally, note we assumed that $\Psi(\cdot)$ is additive in the other items in the basket. {With an additive model, the interaction term $\rho_c^\top \alpha_{c^\prime}$ affects the probability of item $c$ in the same way for each item $c^\prime$ in the basket. In addition, this choice }rules out non-linear interactions with other items. Again, this restriction may be more realistic in some applications than in others, but it is possible to extend the model to consider more complex interaction patterns if necessary. In this paper, we only consider linear interaction effects.
Given (ref), the probability of an ordered basket is the product of the individual choice probabilities. Assuming that $y_{tn_t}$ is the checkout item, it is
The probabilities come from (ref) and (ref), and we have made explicit the dependence on the interaction coefficients and item attributes. The parameters of this softmax are determined by the interaction vectors $\rho_c$ and the attributes $\alpha_c$ of the items that have already been purchased. Given a dataset of (ordered) market baskets, we can use this likelihood to fit each item's latent attributes and interaction.
In many datasets, however, the order in which the items are added to the basket is not observed. \oldtextsc{\scalefont{1.10}shopper}\ implies the likelihood of an unordered set $\mathpzc{y}_t$ by summing over all possible orderings,
Here $\pi$ is a permutation (with the checkout item fixed to the last position) and $\mathbf{y}_{t,\pi}$ is the permuted basket $(y_{t,\pi_1}, \ldots, y_{t,\pi_{n_t}})$. Its probability is in (ref). In (ref), we study a large dataset of unordered baskets; this is the likelihood that we use when fitting \oldtextsc{\scalefont{1.10}shopper}.
Here we describe how the sequential model behind \oldtextsc{\scalefont{1.10}shopper}\ relates to a utility maximization model over unordered sets of items, in which the customer has a specific utility for each basket (i.e., with $2^C$ choices, where $C$ is the number of items).
Let $\mathpzc{y}_t$ be the (unordered) set of items purchased in trip $t$. Define a consumer's utility over unordered baskets as follows:
where $\nu_{c,c'}$ is a term that describes the interaction between $c$ and $c'$. Now consider the following model of utility maximization. At each stage where a consumer must select an item, the consumer has an extreme form of myopia, whereby she assumes that she will immediately check out after selecting this item, and she does not consider the possibility that she could put any items in her basket back on the shelf. Other than this myopia, she behaves rationally, maximizing her utility over unordered items as given by (ref).
The behavioral model of sequential shopping is consistent with the myopic model of utility maximization if (and only if) $\nu_{c,c^\prime}=\rho_c^\top \alpha_{c'} =\rho_{c'}^\top \alpha_c$; this says that the impact of product $c$ on the purchase of product $c'$ is symmetric to the impact of $c$ on product $c'$, holding fixed the rest of the basket. (We do not impose such a symmetry constraint in our model because of the reasons outlined in (ref).) Thus, we can think of utility maximization by this type of myopic consumer as imposing an additional constraint on the probabilistic model. This follows the common practice in economic modeling to estimate the richer models motivated by theory, but without imposing all the restrictions on the parameters implied by that theory; this approach often simplifies computation. browning1991effects also estimate an econometric model without imposing symmetry restrictions implied by utility maximization, and then impose the restrictions in a second step using a minimum-distance approach.
Finally, we note that a fully rational consumer with full information of the price of all products could in principle consider all of the possible bundles in the store simultaneously and maximize over them. However, given that the number of bundles is $2^C$, we argue that considering such a large number of bundles simultaneously is probably not a good approximation to human behavior. Even if consumers are not as myopic as we assume, it is more realistic to assume that they follow some simple heuristics.
The basic \oldtextsc{\scalefont{1.10}shopper}\ model of the previous section captures a customer's sequential choices as a function of latent attributes and interaction coefficients. \oldtextsc{\scalefont{1.10}shopper}\ is flexible, however, in that we can include other forces in the model of customer choice; specifically, they can be incorporated into the (unobserved) mean utility of each item $c$, $\psi_{tc}$, which varies with trip and customer. Here we describe extensions to capture item popularity, customer preferences, price sensitivity, and seasonal purchasing habits (e.g., for holidays and growing seasons). All these factors are important when modeling real-world consumer demand.
We capture overall (time-invariant) item popularity with a latent intercept term $\lambda_c$ for each item. When inferred from data, popular items will have a high value of $\lambda_c$, which will generally increase their choice probabilities.
In our data, each trip $t$ is associated with a particular customer $u_t$, and that customer's preferences affect her choices. We model preference with a per-customer latent vector $\theta_u$. For each choice, we add the inner product $\theta_u^\top \alpha_c$ to the unnormalized log probability of each item. This term increases the probability of types of items that the customer tends to purchase. {Recall that $\theta_u$ is a per-customer variable that appears in every shopping trip $t$ involving customer $u$.} Note that \oldtextsc{\scalefont{1.10}shopper}\ shares the attributes $\alpha_c$ with the part of the model that characterizes interaction effects. The inference algorithm finds latent attributes $\alpha_c$ that interact with both customer preferences and also with other items in the basket.
We next include per-customer price sensitivity. Let $r_{tc}$ denote the price of item $c$ at trip $t$. We consider each customer has an individualized price sensitivity to each item, denoted $\tau_{uc}$, and we add the term $-\tau_{uc} \log r_{tc}$ to the unnormalized log probabilities in (ref). We place a minus sign in the price term to make the choice less likely as the price $r_{tc}$ increases. Further, we constrain $\tau_{uc}$ to be positive; this constraint ensures that the resulting price elasticities are negative. (The price elasticity of demand $\varepsilon$ is a measure used in economics to show the responsiveness of the quantity demanded of a good $y$ to a change in its price $r$; it is defined as $\varepsilon = \partial \log y / \partial \log r$.)
Including $\tau_{uc}$ posits a large number of latent variables (for many customers and items) and, moreover, it is reasonable to assume the sensitivities will be correlated, e.g., a customer's sensitivity to peanut butter and almond butter might be similar. Thus we use a matrix factorization to model the price sensitivities. Specifically, we decompose the user/item price sensitivity matrix into per-user latent vectors $\gamma_u$ and per-item latent vectors $\beta_c$, where $\tau_{uc} = \gamma_{u}^\top \beta_c$. This factorization models the complete matrix of price sensitivities with fewer latent variables.
Finally, instead of the raw price $r_{tc}$, we use the normalized price, i.e., the price for this trip divided by the per-item mean price. Normalized prices have two attractive properties. First, they allow $\beta_c$ to be on a comparable scale, avoiding potential issues that arise when different items have prices that vary by orders of magnitude. Second, they ensure that the other parameters in (ref) capture the average outcome distribution: the price term vanishes when the price takes its average value because the log of the normalized price is zero.
We complete the model with seasonal effects. \oldtextsc{\scalefont{1.10}shopper}\ is designed to estimate the effect of counterfactual changes in policy, such as prices. So, it is important that the parameters of the model associated with price represent the true causal effect of price on customer choices. Seasonal effects are a potential confounder---the season simultaneously affects the price and the demand of an item---and so neglecting to control for seasonality can lead to estimates of the latent variables that cannot isolate the effects of changing prices. For example, the demand for candy goes up around Halloween and the supermarket may decide to put candy on sale during that period. The demand increases partly because of the price, but also because of the time of year. Controlling for seasonal effects isolates the causal effect of price.
We assume that seasonal effects are constant for each calendar week and, as for the price effect, we factorize the week-by-item seasonal effects matrix. (This is motivated by athey2017counterfactual, who conduct a series of empirical studies that support the idea that controlling for week effects is sufficient to identify the causal effect of price.) Denote the week of trip $t$ as $w_t \in \{1, \ldots, 52\}$. We posit per-week latent vectors $\delta_w$ and per-item latent vectors $\mu_c$, and we include a term in (ref) that models the effect of week $w$ on item $c$'s probability, $\delta_w^\top \mu_c$. Note this allows correlated seasonal effects across items; items with similar $\mu_c$ vectors will have similar seasonal effects. So Halloween candy and pumpkin cookies might share similar $\mu_c$ vectors.
We described popularity, preferences, price, and season. We combine these effects in the per-item per-trip latent variable $\psi_{tc}$,
{where the subscripts $u_t$ and $w_t$ indicate, respectively, the customer and week corresponding to shopping trip $t$.} The mean utility $\psi_{tc}$ is used in the probabilities for all $n_t$ choices of trip $t$. With these extensions we have introduced several new latent variables to infer from data: item popularities $\lambda_c$, customer preferences $\theta_u$, price sensitivity factorizations $\gamma_u$ and $\beta_c$, and seasonal factorizations $\delta_w$ and $\mu_c$.
As a final extension, we develop a model of customers that “think ahead.” Consider a pair of items $c$ and $c'$, where both $\rho_c^\top \alpha_{c'}$ and $\rho_{c'}^\top \alpha_{c}$ are high, so that the value of $c$ increases when $c'$ is in the basket and vice versa. In this case, we say that the goods are complements to the consumer. Accurately accounting for complementarity is particularly important when estimating counterfactuals based on different prices; theoretically increasing the price of one item should lower the probability of the other.
In this scenario, our baseline model specifies that the consumer will consider the goods one at a time. The effect of a high price for item $c'$ is that it reduces the probability the item is chosen (and if chosen, it is less likely to be chosen early in the ordering). Not having item $c'$ in the basket at each new consideration event reduces the attractiveness of item $c$. However, a more realistic model has the consumer anticipate when she considers item $c$ that she might also consider item $c'$. When the consumer considers the pair, the high price of one item deters the purchase of both.
We address this issue by introducing thinking ahead. When considering the item at step $i$, the consumer looks ahead at the $i+1$th step the customer may purchase. We emphasize that the model does not, at this point, assume that the consumer actually purchases the next item according to that rule. When the consumer comes to consider what to choose for step $i+1$, she follows the same think-ahead-one-step rule just described.
Thinking ahead adjusts the unnormalized log probabilities of the current item in (ref). Specifically, it adds a term to the log linear model that is the utility of the optimal next item. (Note this could be the checkout item.) The unnormalized log probability is
where $[\mathbf{y}_{t,i-1}, c]$ is a hypothetical basket that contains the first $i-1$ purchased items and item $c$. Since the customer has in mind a tentative next item when deciding about the current item, we call this “thinking one step ahead.”
Note that thinking ahead assumes that the item $c^\prime$ will itself be selected without looking further ahead, that is, it is chosen purely based on its own utility without accounting for the interaction effects of $c^\prime$ with future purchases. The thinking-ahead idea can be extended to two or more steps ahead, but at the expense of a higher computational complexity. See (ref) for an illustrative example that clarifies how the thinking-ahead procedure works.
The thinking-ahead mechanism is an example of inferring agent's preferences from their behavior in a dynamic model, a type of exercise with a long history in applied econometrics (e.g. wolpin1984estimable, hotz1993conditional). From the machine learning perspective, it closely resembles the motivation behind inverse reinforcement learning Russell1998,Ng2000. Inverse reinforcement learning analyzes an agent's behavior in a variety of circumstances and aims to learn its reward function. (This is in contrast to reinforcement learning, where the reward function is given.) To see the connection, consider the unnormalized log probability of (ref) as the reward for choosing $c$ and then acting optimally one step (i.e., choosing the best next item). With this perspective, fitting \oldtextsc{\scalefont{1.10}shopper}\ to observed data is akin to learning the reward function.
Finally, note the thinking-ahead model also connects to utility maximization over unordered baskets, with the utility function given in (ref). Now the model is consistent with a consumer who maximizes her utility over unordered items at each step, but where at each step she is myopic in two ways. First, as before, she does not consider that she can remove items from her cart. Second, she assumes that she will buy exactly one more item in the store and then check out, and the next item will be the one that maximizes her utilty over unordered baskets (taking as given her cart at that point, including the latest item added, as well as the belief that she will then check out). Again, our probabilistic model is consistent with this interpretation if and only if $\rho_{c'}^\top \alpha_c = \rho_{c}^\top \alpha_{c'}$ for all $(c,c')$ {(a constraint that we do not enforce)}.
We specified \oldtextsc{\scalefont{1.10}shopper}, a sequential model of shopping trips. Each trip is a tuple $(\mathbf{y}_t, u_t, w_t, r_t)$ containing a collection of purchased items ($\mathbf{y}_t$), the customer who purchased them ($u_t$), the week that the shopping trip took place ($w_t$), and the prices of all items ($r_t$). The likelihood of each shopping trip captures the sequential process of the customer choosing items ((ref)), where each choice is a log linear model ((ref)). \oldtextsc{\scalefont{1.10}shopper}\ includes terms for item popularity, customer preferences, price sensitivity, and seasonal effects ((ref)). We further described a variant that models the customer thinking ahead and acting optimally ((ref)). Though this is a model of ordered baskets, we can use it to evaluate the probability of an unordered basket of items ((ref)).
\oldtextsc{\scalefont{1.10}shopper}\ is based on log linear terms that involve several types of latent parameters: per-item interaction coefficients $\rho_c$, per-item attributes $\alpha_c$, per-item popularities $\lambda_c$, per-user preferences $\theta_u$, per-user per-item price sensitivities $(\gamma_u, \beta_c)$, and per-week per-item seasonal effects $(\delta_w, \mu_c)$. Our goal is to estimate these parameters given a large data set of $T$ trips $\{(\mathbf{y}_t, u_t, w_t, r_t)\}_{t=1}^{T}$.
We take a Bayesian approach. We place priors on the parameters to form a Bayesian model of shopping trips, and then we form estimates of the latent parameters by approximating the posterior. We use independent Gaussian priors for the real-valued parameters, $\rho_c$, $\alpha_c$, $\theta_u$, $\lambda_c$, $\delta_{w}$, and $\mu_c$. We use gamma priors for the positive-valued parameters associated with price sensitivity, $\gamma_u$ and $\beta_c$. With the approximate posterior, we can use the resulting estimates to identify various types of purchasing patterns and to make predictions about the distribution of market baskets; in particular predictions under price changes.
{Finally, note that }\oldtextsc{\scalefont{1.10}shopper}{ uses matrix factorization approaches for customer preferences ($\theta_u^\top \alpha_c$), price effects ($\gamma_u^\top \beta_c$), and seasonal effects ($\delta_w^\top \mu_c$). Matrix factorization has two sources of invariance (label switching and rescaling). This implies that we cannot uniquely identify individual variables such as $\theta_u$ or $\gamma_u$ from the data. Fortunately, this is not an issue for (counterfactual) predictions, because they only depend on the inner products and not on the individual coefficients.}
We now describe a simulation of customers to illustrate how \oldtextsc{\scalefont{1.10}shopper}\ works. The purpose of our simulation study is to show \oldtextsc{\scalefont{1.10}shopper}'s ability to handle heterogenous purchases, complements, and counterfactual settings (i.e., interventions on price). More specifically, we ask whether the model can disentangle correlated preferences and complements, both of which might contribute to co-purchasing. We illustrate that distinguishing between these two types of relationships relies on the thinking ahead property from (ref).
We simulate the following world.
With this specification, we simulate $100$ different customers ($50$ new parents and $50$ college students) and $1000$ trips per customer. For each trip, the first four items ({\it coffee}, diapers, ramen, candy) each have a $40\%$ chance of being marked up to a high price. Further, there is a $60\%$ chance that one of the items in the complementary pairs (hot dogs, hot dog buns, taco shells, \textit{taco seasoning}) is marked up. At most one of the items in the complementary pairs has a high price.
Given the simulated data, we fit a \oldtextsc{\scalefont{1.10}shopper}\ model that contains terms for latent attributes, user preferences, and price. For simplicity, we omit seasonal effects here. (The approximate inference algorithm is described below in (ref).) One of our goals is to confirm the intuition that thinking ahead correctly handles complementary pairs of items. Thus we fit \oldtextsc{\scalefont{1.10}shopper}\ both with and without thinking ahead.
Consider a shopping trip where the prices of {\it coffee}\ and taco shells\ are high, and consider a customer who is a new parent. As an example, this customer may buy diapers, hot dogs, hot dog buns, and then check out. (ref) shows the {predicted} probabilities of each possible item at each stage of the shopping trip. They are illustrated for the models with and without thinking ahead.
First consider the preference items in the first stage, before anything is placed in the basket. The customer has a high probability of purchasing diapers, and low probabilities of purchasing {\it coffee}, ramen, and candy. The probabilities for ramen\ and candy\ are low because of the type of customer (new parent); the probability for {\it coffee}\ is low because of its high price.
Now consider the complementary pairs. In the model without thinking ahead, the customer has high probability of buying hot dogs, hot dog buns, and taco seasoning; because of its high price, she has a low probability of buying taco shells. But this is incorrect. Knowing that the price of taco shells\ is high, she should have a lower probability of buying taco seasoning\ because it is only useful to buy along with \textit{taco shells}. The thinking-ahead model captures this, giving both \textit{taco seasoning}\ and \textit{taco shells}\ a low probability.
Subsequent stages further illustrate the intuitions behind \oldtextsc{\scalefont{1.10}shopper}. First, each stage zeros out the items that are already in the basket (e.g., at stage 2, diapers\ have probability $0$). Second, once one item of the complementary pair is bought, the probability of the other half of the pair increases and the probabilities of the alternative pair becomes low. In this case, once the customer buys hot dogs, the probability of the taco products goes to zero and the probability of hot dog buns\ increases.
As a final demonstration on simulated data, we generate a test set from the simulator with $30$ shopping trips for each customer. On this test set, we “intervene” on the price distribution: the probability of a preference item having a high price is $0.95$ and one of the four complementary items always has a high price. On this test set, a better model will provide higher held-out log probability and we confirm that thinking ahead helps. The thinking-ahead model gives an average held out log probability of $-2.26$; the model without thinking ahead gives an average held out log probability of $-2.79$.
\oldtextsc{\scalefont{1.10}shopper}\ relates closely to several lines of research in machine learning and economics. We discuss them in turn.
One of the central ideas in \oldtextsc{\scalefont{1.10}shopper}\ is that the items in the store have latent vector representations and we estimate these representations from observed shopping basket data. In developing \oldtextsc{\scalefont{1.10}shopper}, we were directly inspired by the neural probabilistic language model of Bengio2003,bengio2006neural. That model specifies a joint probability of sequences of words, parameterized by a vector representation of the vocabulary. In language, vector representations of words (also called “distributed representations”) allow us to reason about their usage and meaning Harris1954,Firth1957,Bengio2003,Mikolov2013distributed. Here we expand on this idea to study consumer behavior, and we show how the same mathematical concepts that inspired word representations can help understand large-scale consumer behavior.
We note that neural language models have spawned myriad developments in other so-called word embedding methods for capturing latent semantic structure in language Mnih2007,Mnih2012,Mikolov2013efficient,Mikolov2013distributed, Mikolov2013linguistic,Mnih2013learning,Pennington2014,Levy2014neural, Vilnis2014,Arora2016,Barkan2016bayesian,Bamler2017. There has been some work on extrapolating embedding ideas to shopping items Rudolph2016,Liang2016,Barkan2016, but these newer methods are not readily appropriate to the analyses here. The reason is that they are conditionally specified models, defined by the conditional distribution of each item given the others. In contrast, \oldtextsc{\scalefont{1.10}shopper}\ directly models the joint distribution of items; this enables \oldtextsc{\scalefont{1.10}shopper}\ to more easily account for additional information, such as price and customer preferences. From the econometric perspective, casting \oldtextsc{\scalefont{1.10}shopper}\ as a proper generative model enables a direct interpretation as a structural equation model with meaningful parameters. This lets us use \oldtextsc{\scalefont{1.10}shopper}\ to make counterfactual predictions, such as about how price changes will affect firm revenue and consumer welfare.
The way \oldtextsc{\scalefont{1.10}shopper}\ models customer preferences is similar to modern recommender systems, where matrix factorization models are standard. Matrix factorization decomposes observed data of customer/item interactions into latent preferences and item attributes Canny2004,Hu2008,Ma2011,Wang2011,Gopalan2015, {potentially with an infinite number of latent parameters }Gorur2006,Gopalan2014{. Matrix factorization methods have been used for web services recommendations }Stern2009{ and also market basket data }Wan2017. Though it is effective, matrix factorization does not directly capture item-to-item co-occurrence, which is the main motivation behind models based on latent attributes. \oldtextsc{\scalefont{1.10}shopper}\ includes both attributes and preferences, thereby combining ideas from distributed representations with those from recommendation. \oldtextsc{\scalefont{1.10}shopper}\ also goes further by including price (and price sensitivities) and seasonal effects, and generally seeking realistic model of consumer behavior. It is more appropriate than machine learning recommendation methods for evaluating counterfactual questions. {Incorporating observed attributes also relates to recommender systems with attributes }Abernethy2009{, although }\oldtextsc{\scalefont{1.10}shopper}{ can consider time-varying item attributes such as prices.}
\oldtextsc{\scalefont{1.10}shopper}\ expands on discrete choice models of consumer behavior, models widely used in economics and marketing. The majority of papers in this literature analyze the discrete choice of a consumer who selects a single product from a set of prespecified imperfect substitutes; examples include choices of laundry detergent, personal computers, or cars. As reviewed by keane2013panel, this literature focuses on estimating cross-price elasticities, accounting for (latent) consumer heterogeneity in tastes. A small literature on “market mapping” elrod1988choice,elrod1995factor,chintagunta1994heterogeneous considers latent attributes of items within a product category (e.g., laundry detergent). \oldtextsc{\scalefont{1.10}shopper}\ is similar in spirit in its focus on estimating latent characteristics of items using panel data. However, \oldtextsc{\scalefont{1.10}shopper}\ differs in the scale it handles, considering hundreds or thousands of items and hundreds of thousands of shopping trips. This is not simply a matter of improving computational speed; analyzing the whole market for counterfactual prediction requires a more thorough treatment of issues like complementarity, substitutability, and customer heterogeneity. {A complementary approach to ours is the work by }Semenova2018{, who consider observational high-dimensional product attributes (e.g., text descriptions and images) rather than latent features.} Also see our related paper for another approach at this same scale athey2017counterfactual.
\oldtextsc{\scalefont{1.10}shopper}\ also differs significantly from other discrete choice models in that it considers interaction effects among a large number of items in a consumer's basket without a priori imposing a structure of product relationships. A variety of datasets track a panel of consumers over time, recording their baskets in each shopping trip. This includes supermarket shopping datasets and online panel datasets of shopping from vendors like Neilsen and comScore, as well as data collected internally by online and offline retailers. However, there is relatively little work in economics and marketing that analyzes consumers' choices over many items in a basket using this type of data. Even when multiple items are considered, few methods attempt to estimate directly whether items are substitutes or complements; instead, products are typically assumed to be independent of one another across categories, or strong substitutes within categories (in the sense that purchasing one item precludes purchasing others). \oldtextsc{\scalefont{1.10}shopper}\ does not require a prespecified ontology of categories; the interaction effects are estimated directly from the data.
Of course, estimating the extent to which products are substitutes and complements for one another is a hard problem when there are many products: if there are $C$ products, there are $2^C$ baskets, and without further restrictions, a consumer's preferences over baskets thus has $2^C$ parameters. One approach is to make use of more aggregate data combined with functional form assumptions; for example, the almost ideal demand system deaton1980almost considers a consumer maximizing utility subject to a budget constraint over a period of time, where the share of budget allocated to each good is taken to be continuous. With functional form assumptions, the consumer's expenditure share for each product can be written as a linear function of transformations of the prices of other products, and the parameters of these demand functions can be interpreted as expressing underlying consumer preferences, including the extent to which products are substitutes or complements. This approach has a number of attractive features, including tractability and computational simplicity, and it can be used to handle many products simultaneously. However, in reality prices often change relatively frequently, and without data about when a consumer shopped or what prices the consumer saw, inferences drawn from aggregate data can be misleading. In addition, more aggregated data loses crucial information about the co-purchases an individual makes on the same trip.
A small set of papers in economics and marketing athey1998empirical attempts to use individual choice data estimate parameters that describe interaction effects (substitutes or complements) with a very small number of items; see chintagunta2011structural and berry2014structural for recent surveys of the literature. For example, train1987demand treat each bundle as a discrete alternative, but use nested logit to account for correlation among related bundles, while gentzkow2007valuing incorporates a parameter in the utility for complementarity among two items. song2007discrete build a utility-maximization framework where consumers select not just whether to purchase, but how much, and apply it to supermarket purchase data for two products, laundry detergent and fabric softener. However, due to computational considerations, most of the papers that deal with disaggregated data have focused on a very small number of products and a small number of customers. With \oldtextsc{\scalefont{1.10}shopper}, we are able to jointly model thousands of items and millions of purchased items.
(ref) defined \oldtextsc{\scalefont{1.10}shopper}, a Bayesian model of market baskets. Given a large data set, our goal is to infer the latent parameters, the components that govern item attributes, user preferences, price sensitivity, and seasonal effects. We then use the inferred parameters to make predictions, e.g., about demand, and to characterize patterns of purchase behavior. In this section we describe a variational inference algorithm to solve this problem.
For conciseness, we denote the collection of latent variables as $\mb{\ell} = \{\rho,\alpha, \lambda, \theta, \gamma, \beta, \mu, \delta\}$, the observed shopping baskets as $\mathpzc{y}=\{\mathpzc{y}_t\}$, and the observed characteristics of shopping trips as $\mathbf{x} = x_{1:T}$, where $x_t = (u_t, w_t, r_t)$ indicates the customer, calendar week, and item prices for the $t$th trip. The posterior is
It is difficult to compute the posterior in closed form because of the intractable denominator, the marginal likelihood of the observed data. Further, as we described in (ref), the likelihood term is itself intractable because of the sum over the large number of items in the denominator of the softmax function. We must use approximate Bayesian inference.
Variational inference Jordan1999,Wainwright2008 is an alternative to MCMC for approximating otherwise intractable Bayesian posteriors. Compared to MCMC, it more easily scales to large data, and especially so in non-conjugate models such as \oldtextsc{\scalefont{1.10}shopper}.
Variational inference approximates the posterior with a parameterized family of distributions of the latent variables $q(\mb{\ell} ; \nu)$; the parameters $\nu$ are called variational parameters. The idea is to find the member of the family that is closest to the exact posterior, where closeness is measured by the \gls{KL} divergence. Minimizing the \gls{KL} divergence is equivalent to maximizing the \gls{ELBO},
where the expectations are taken with respect to the variational distribution.
The \gls{ELBO} $\mathcal{L}(\nu)$ provides a lower bound on the marginal likelihood $\log p(\mathpzc{y} \, | \, \mathbf{x})$, and hence its name. Its maximizer is $\nu^* = \arg \max_{\nu} \mathcal{L}(\nu)$ (which also minimizes the \gls{KL} divergence). When we use variational inference, we first optimize the \gls{ELBO} and then use the resulting distribution $q(\mb{\ell} ; \nu^*)$ as a proxy for the exact posterior, e.g., to investigate the inferred hidden structure or to approximate the posterior predictive distribution. In summary, variational inference uses optimization to approximate the posterior. For a review of variational inference, see Blei:2016.
To develop a variational inference algorithm for \oldtextsc{\scalefont{1.10}shopper}, we first specify the variational family and then develop the optimization procedure for fitting its parameters. (ref) Ruiz2017_supp gives all the details; here we sketch the main ideas.
The first step to deriving a variational inference algorithm is to specify the variational family $q(\mb{\ell} ; \nu)$. Following many successful applications of variational inference, we posit the mean-field family, where each latent variable is governed by its own distribution and endowed with its own variational parameter. We use Gaussian variational factors for the latent variables with Gaussian priors; and gamma variational factors for the latent variables with gamma priors. Though the mean-field variational family makes strong independence assumptions, we emphasize that latent variables are not identically distributed and thus the variational family is still very flexible. Thus, for example, the variational distribution of the latent attributes of peanut butter will peak at one point in attribute space; the latent attributes of taco shells will (likely) peak at a different point.
Given the variational family, we next formulate how to optimize (ref) with respect to the variational parameters; we use gradient-based stochastic optimization. This problem is complicated for several reasons: the data set sizes are large, the expectations are intractable, and the likelihood is expensive to calculate. To overcome these obstacles, we bring together a variety of recent innovations around variational inference, summarized below and detailed in the supplement Ruiz2017_supp.
See \url{https://github.com/franrruiz/shopper-src} for a publicly available implementation of \oldtextsc{\scalefont{1.10}shopper}.
We now use \oldtextsc{\scalefont{1.10}shopper}\ to study a real-world data set of consumers. Our dataset contains $570{,}878$ baskets from an large grocery store (anonymized for data privacy reasons) Che2012. The dataset is available to researchers at Stanford and Berkeley by application; it has been used previously in other research papers (see \url{https://are.berkeley.edu/SGDC}). These baskets contain $5{,}968{,}528$ purchases of $5{,}590$ unique items. The data spans $97$ weeks and $3{,}206$ different customers. We split the baskets into {a training, validation, and test set. The test set contains all the baskets in the last $2$ months of the data collection period; the validation set contains $5\%$ of the remaining purchases, chosen at random.}
We study the predictive performance of \oldtextsc{\scalefont{1.10}shopper}\, both observationally and under price intervention. Note that the store changes prices regularly, and this variation allows us to empirically separate correlated latent preferences from price effects, once we control for seasonality.
We also demonstrate how to use \oldtextsc{\scalefont{1.10}shopper}\ to qualitatively analyze the data. It finds interpretable latent features, which can be used to find complementary and exchangeable items.
We first focus on category-level data, where we group individual items---according to their \gls{UPC}---into their category (such as “oranges,” “flour,” “pet supplies,” etc.); there are $374$ unique categories. The category-level data contains fewer items and is computationally less demanding for any model; so we use these data to compare the predictive performance of \oldtextsc{\scalefont{1.10}shopper}\ to other models. We obtain better predictive accuracy (in terms of test log-likelihood) than the competing methods, especially under price interventions.
Secondly, we focus on \gls{UPC} data, all $5{,}590$ items. Based only on posterior inference with the observed basket data, \oldtextsc{\scalefont{1.10}shopper}\ identifies characteristics of items, including complements and exchangeable items. We define quantities to measure these concepts, and we find that exchangeability is a good proxy to find substitutes.
We compare \oldtextsc{\scalefont{1.10}shopper}\ to other latent factor models of shopping data. In particular, we apply \gls{HPF} Gopalan2015 and exponential family embeddings Rudolph2016. \gls{HPF} focuses on user preferences; exponential family embeddings focuses on item-to-item interactions. (ref) summarizes the differences between \oldtextsc{\scalefont{1.10}shopper}\ and these models; note that none of the comparison models captures price or seasonal effects.
{We also fitted two simpler models. First, we studied a multinomial logistic regression model that predicts an item conditioned on the rest of the items in the basket and the item prices. The inputs are binary indicators of the items in the basket excluding the target item, as well as the logarithm of the normalized prices of all the items. Second, we studied factor analysis }Cattell1952{ on the basket-by-item matrix. However, we do not report the results of these two models in the paper because they gave worse predictions than all other methods on the test set, corresponding to the last two months of data.} (On the validation set, the multinomial logistic regression model performed slightly better than exponential family embeddings but worse than \gls{HPF}. We emphasize that multinomial logistic regression takes the item prices as inputs, in contrast to exponential family embeddings and \gls{HPF}.)
We fit each model to the category-level data. In \oldtextsc{\scalefont{1.10}shopper}\, we set most of the Gaussian hyperparameters to zero mean and unit variance. (The seasonal effect hyperparameters have a smaller variance, 0.01, because we do not expect large seasonal effects.) The price sensitivity parameters have Gamma priors; we use a shape of $1$ and a rate of $10$. As for the comparison models, we modify \gls{HPF} to allow for multiple shopping trips of the same user; in its original construction it can only capture a single trip. Finally, we implement two versions of exponential family embeddings, Bernoulli embeddings for binary data (labeled “B-Emb”) and Poisson embeddings for count data (labeled “P-Emb”). We weight the zeros by a factor of $0.1$, as suggested by Rudolph2016. For all methods, we use the validation set to assess convergence.
{To choose the number of latent factors $K$, we first set the number of factors of the price and seasonal vectors to $10$ and run }\oldtextsc{\scalefont{1.10}shopper}{ with $K\in\{10,20,50,100,200\}$. We choose the value $K=100$ because it provides the best predictions on the validation set. For that value of $K$, we then explore the number of latent factors for the price and seasonal vectors in the set $\{5,10,20,50\}$, finally choosing $10$. We also set $K=100$ for }\gls{HPF}{ and exponential family embeddings.}
To evaluate the models, we calculate the average log-likelihood of the test-set items. For each, we calculate its probability conditioned on the other observed items in the basket. Higher log probability indicates a better model fit. (ref) shows the results. {The numbers in parentheses show the standard deviation, obtained by using bootstrap on the test samples.}
We study several types of test sets. The second column corresponds to a typical test set, {containing two months of data}. Columns 3-5 focus on skewed test sets, {where the target items have more extreme prices with respect to their average price within each month (outside of the range $\pm 2.5\%$, $\pm 5\%$, and $\pm 15\%$). To control for seasonal effects, we consider items whose price is outside that range with respect to the per-month average price.} These evaluations are suggestive of the performance under price intervention, i.e., where the distribution of price is different in testing than it is in training. The numbers in parentheses {on the table heading} indicate the number of purchases considered in each column.
We report results that incrementally add terms to the basic \oldtextsc{\scalefont{1.10}shopper}\ model. The most basic model contains user information with item-to-item interactions (“I+U”); it improves predictions over the competing models. {The next model includes seasonal effects (“I+U+S”); it improves performance on the typical test set only.} (To compute predictions for the models with seasonal effects, we set the seasonal parameter $\delta_w$ for the weeks in the test set equal to the value of $\delta_w$ corresponding to the same week but the year before.) {We also consider a model that} includes price sensitivity (“I+U+P”); it further improves performance. The full model adds seasonal effects (“I+U+P+S”); {in general} it gives the best performance. As expected, modeling price elasticity is important in making counterfactual predictions. The performance gap between the models with and without price increases as the test-set prices diverge from their average values.
(ref){ compares models with different numbers of latent parameters. (}(ref){ gives the number of latent parameters in each model.) Exponential family embeddings posit two $K$-length vectors for each item. }\gls{HPF}{ has one $K$-length vector for each user and item. }\oldtextsc{\scalefont{1.10}shopper}{ has one $K$-length vector for each user and two $K$-length vectors for each item, in addition to the intercept terms and the price and seasonal components.}
Finally, we study the empirical performance of “thinking ahead,” particularly when predicting groups of items. (ref) shows two metrics: the first column is the average (per-item) test log-likelihood over three items, conditioned on the rest of items in the basket; the second column is average (per-item) log-likelihood over the entire basket. (Here we exclude the checkout item{, and we compute the predictions based on the ordering in which items are listed in the test set}.) The model with “thinking ahead” more correctly handles complements and price sensitivity, and it provides better predictive performance.
\oldtextsc{\scalefont{1.10}shopper}\ provides a better predictive model of consumer behavior. We now use the fitted model to qualitatively understand the data.
First, we assess the attributes vectors $\alpha_c$, confirming that they capture meaningful dimensions of the items. (Recall that each is a 100-dimensional real-valued vector.) As one demonstration, we project them on to 2-dimensional space using \gls{tSNE} vanderMaaten2008, and then examine the items in different regions of the projected space. (ref) shows two particular regions: one collects different types {of pet food and supplies}; the other collects different cleaning products. As a second demonstration, we can use the cosine distance to find similar items similar to a “query item.” (ref) shows the top-three most similar items to a set of queries.
Other latent variables reveal different aspects of consumer behavior. For example, (ref) show the highest and lowest seasonal effects for a set of items. The model correctly captures how Haloween candy is more popular near Halloween; and turkey is more popular near Thanksgiving. It also captures the seasonal availability of fruits, e.g., cherries.
These investigations are on the category-level analysis. For more fine-grained qualitative assessments---especially those around complementarity and exchangeability---we now turn to the \gls{UPC}-level model.
We fit \oldtextsc{\scalefont{1.10}shopper}\ to \gls{UPC}-level data, which contains $5$,$590$ unique items. {We use the same dimensionality of the latent vectors as in }(ref){, i.e., $K=100$ for $\alpha_c$, $\rho_c$, and $\theta_u$, and $10$ latent features for the seasonal and price vectors. We additionally} tie the price vectors $\beta_c$ and seasonal effect vectors $\delta_c$ to all items in the same category. To speed up computation, we fit this model without thinking ahead.
We can again find similar items to “query” items using the cosine distance between attribute vectors $\alpha_c$. (ref) shows similar items for several queries; the model identifies qualitatively related items.
For another view, (ref) shows a two-dimensional \gls{tSNE} projection vanderMaaten2008 of the attribute vectors. This figure colors the items according to their group, and it reveals that items in the same category are often close to each other in attribute space. (Groups are defined as one level of hierarchy above categories; some examples are “jams, jellies, and spreads,” “salty snacks,” or “canned fruits.”) When groups are mixed in a region, they tend to be items that appear in similar contexts, e.g., hot dogs, hamburger buns, and soda ((ref)).
A key objective for applications of \oldtextsc{\scalefont{1.10}shopper}\ is to be able to estimate interaction effects among products. These effects are described by the coefficients $\rho_c$ and attributes $\alpha_c$. When $\rho_c^\top \alpha_{c'}$ and $\rho_{c'}^\top \alpha_c$ are large, this means that purchasing item $c^\prime$ increases the consumer's preference for $c$, and vice-versa. When these two terms are negative and large, the items may be substitutes---putting one item in the basket reduces the need for the other item.
All else equal, complements are relatively likely to be purchased together, while substitutes are less likely to be purchased together. In addition, for complementary items $c$ and $c'$, when the price of $c'$ increases, the customer is less likely to purchase item $c$. We define the complementarity metric as
Most shopping data is sparse---each individual item has a low probability of purchase---and so it can be difficult to accurately identify pairs of items that are substitutes. We introduce a new metric, which we call exchangeability, that can help identify substitutes. For a pair of items $(c,c')$, our notion of exchangeability depends on the distributions of items that are induced when conditioning on $c$ or $c'$; if those distributions are similar then we say that $c$ and $c'$ are exchangeable.
Let $p_{k|c}$ denote the probability of item $k$ given that item $c$ is the only item in the basket (in this definition, we zero-out the probabilities of items $c$, $c'$, and checkout). We measure the similarity between the distributions $p_{\cdot|c}$ and $p_{\cdot|c'}$ with symmetrized \gls{KL} divergence. The exchangeability metric is
With this definition, two items that are intuitively exchangeable---such as two different prepared sandwiches, or two yogurts that are the same brand---will exhibit smaller values of $E_{cc'}$.
With several example queries, (ref) shows the three most complementary items and the three most exchangeable items. In the computation of the probabilities $p_{k|c}$, we consider an “average customer” (obtained by averaging the per-user vectors) in an “average week” (obtained by averaging the per-week vectors). The probabilities were also obtained assuming that no other item except $c$ is in the shopping basket. \oldtextsc{\scalefont{1.10}shopper}\ correctly recovers complements, such as taco shells and taco seasoning, hot dogs and buns, and mustard and hamburger buns. \oldtextsc{\scalefont{1.10}shopper}\ typically identifies the most exchangeable items as being of the same type, such as several types of buns or tortillas; highly exchangeable items are suggestive of substitutes.
{Finally, note that }\oldtextsc{\scalefont{1.10}shopper}{ can model triplets of complementary items when they are also pairwise complements, but it cannot in cases where the triplets do not form complementary pairs.}
We developed \oldtextsc{\scalefont{1.10}shopper}, a probabilistic model of consumer behavior. \oldtextsc{\scalefont{1.10}shopper}\ is a sequential model of a consumer that posits a latent structure to how he or she shops. In posterior inference, \oldtextsc{\scalefont{1.10}shopper}\ estimates item attributes, item-item interactions, price elasticities, seasonal effects, and customer preferences. We used \oldtextsc{\scalefont{1.10}shopper}\ to analyze large-scale shopping data from a grocery store and evaluated it with out-of-sample predictions. In addition to evaluating on a random sample of held out data, we also evaluated on counterfactual test sets, i.e., on days where prices are systematically different from average.
\oldtextsc{\scalefont{1.10}shopper}\ is a computationally feasible, interpretable approach to modeling consumer choice over complex shopping baskets. While the existing literature in economics and marketing usually considers two or three products berry2014structural, \oldtextsc{\scalefont{1.10}shopper}\ enables considering choices among thousands of products that potentially interact with one another. Using supermarket shopping data, we show that \oldtextsc{\scalefont{1.10}shopper}\ can uncover these relationships.
There are several avenues for future work. Consider the question of how the probabilistic model relates to maximizing a consumer's utility of the entire basket. (ref) introduced a heuristic model of consumer behavior that is consistent with the probabilistic model. This heuristic lends itself to computational tractability; it enables us to analyze datasets of baskets that involve thousands of products and thousands of consumers. But the heuristic also involves a fairly myopic consumer. In future work, it is interesting to consider alternative heuristic models.
Another avenue is to embellish the distribution of baskets. One way is to expand the model to capture within-basket heterogeneity. Shopping trips may reflect a collection of needs, such as school lunches, a dinner party, and pet care, and items may interact with other items within each need. Capturing the heterogenous patterns within the baskets would sharpen the estimated interactions. {Another embellishment is to expand the model of baskets to include a budget. A budget imposes constraints on the number of items (or their total price) purchased in a single trip.}
Francisco J.\ R.\ Ruiz is supported by the EU H2020 programme (Marie Sk\lodowska-Curie Individual Fellowship, grant agreement 706760). This work is also supported by ONR N00014-17-1-2131, NIH 1U01MH115727-01, DARPA SD2 FA8750-18-C-0130, ONR N00014-15-1-2209, NSF CCF-1740833, IBM, 2Sigma, and Amazon. The authors also thank Tilman Drerup and Tobias Schmidt for research assistance, the seminar participants at Stanford, as well as the Cyber Initiative at Stanford. Finally, we also acknowledge the support of NVIDIA Corporation with the donation of a GPU used for this research.