Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
23,818 characters · 15 sections · 19 citation commands
Estimating Supply Incrementality in Two-sided Marketplaces: A Causal Machine Learning Approach
\email{[email removed]}
\email{[email removed]}
\email{[email removed]}
\begingroup\footnote{\textcopyright\ 2025 Airbnb, Inc. All rights reserved.}\addtocounter{footnote}{-1}\endgroup
Two-sided marketplaces face the fundamental challenge of balancing supply and demand across heterogeneous product segments. This is critical for marketplaces to optimize product features to better serve both sides of the marketplace. For example, Airbnb provides tools that help hosts to choose and set competitive prices, in order to best meet the demand from guests (ye2018customized).
As a critical piece of the general puzzle for balancing supply and demand, it is important to understand the incremental value of additional supply for different listing segments. We will refer to this as supply incrementality. An Airbnb blog post by sanchez2025airbnb discusses supply incrementality as part of estimating the lifetime value for each listing. The Venn diagram in their Figure 1 illustrates the concept of supply incrementality, or by how much certain listings actually grow total bookings instead of cannibalizing bookings from pre-existing listings. The pink circle represents total bookings with the pre-existing listings, while the blue circle represents bookings with certain new listings. The overlapping section represents cannibalization — bookings that shifted from other listings to the new supply — while the non-overlapping portion of the blue circle reflects the new bookings that are generated by the new supply. Higher supply incrementality of a listing segment means there had been insufficient supply to meet demand, and the additional supply helped meet the demand from guests.
sanchez2025airbnb also noted examples of why such work helps serve the community more effectively, such as pinpointing listing segments that have an opportunity to get more bookings and identifying which marketing initiatives bring the most value to the community.
As important as it is to measure supply incrementality, it presents significant challenges. The core challenge is that we don’t observe the counterfactual: what would have happened in the marketplace if certain supply had not been available — it’s not practical to run experiments that randomize supply availability, and observational data is confounded by endogeneity and substitution effects.
To address these challenges, we combine a causal machine learning approach and a hierarchical Bayesian framework, incorporating measures of product similarity based on the geospatial literature.
We find that such models are able to provide plausible estimates of the marketplace returns to additional supply. Even though our discussion is focused on the Airbnb marketplace, the methodology we develop has broad applications to other two-sided marketplaces where understanding the causal impact of supply changes is valuable.
sanchez2025airbnb laid the groundwork for estimating supply incrementality at Airbnb, which is based on a Cobb-Douglas model. The Cobb-Douglas model is a widely used production function that relates output to the inputs of capital and labor (douglas1976cobb) and has been used more recently as a matching function for matching workers with job vacancies (petrongolo2001looking). In the context of marketplace supply incrementality, it models the relationship between supply, demand, and marketplace outcomes (bookings) with constant outcome elasticities for each input. The referenced blog post applies this model to estimate how increases in supply of listings affect total bookings, with incrementality parameters indicating the percentage change in bookings for a percentage change in supply.
There are also other structural models of demand and supply in marketplaces with heterogeneous products. For example, as gandhi2021empirical summarizes, structural models typically employ discrete choice frameworks like Berry, Levinsohn, Pakes (BLP) models (berry1995automobile) or random coefficient logit models to capture consumer preferences across differentiated products. These models often specify utility functions, impose assumptions about substitution patterns, and estimate demand parameters using methods like generalized method of moments.
Our approach imposes less strict functional form assumptions on the relationship between supply, demand, and marketplace outcomes. We take this approach for several key reasons:
The Double ML approach works by first predicting both supply and bookings using other features in the first stage, and then regressing residual bookings on residual supply in the second stage. Intuitively, we try to focus on the causal impact of supply on bookings, after accounting for the potential influence of other factors such as demand-supply tightness, listing attributes, etc.
Let $g$ denote a broad grouping of listing segments, $j$ denote listing segments, and $t$ denote time period. We construct the following metrics at the product segment $\times$ time period level, from which we can construct metrics to estimate the model. The broad grouping of listing segments ($g$) is defined in a way to minimize cannibalization, the same as the listing segments considered by sanchez2025airbnb. However, for more granular listing segments ($j$), there could be overlap in demand across listing segments, making it important to consider the substitution between different listing segments.
Let $Y$ denote the outcome, bookings, $S$ denote treatment, supply, and $X$ denote covariates, including demand patterns, listing features, trend, and seasonality. In the first stage of the Double ML model, we use all the covariates to help explain the observed variation in supply and bookings, so we can focus on the remaining unexplained variation for the second stage. In other words, we first predict $Y$ and $S$ based on the same $X$ on the right hand side, so we can later estimate the impact of residual $S$ on residual $Y$, isolating the impact of supply on bookings, holding everything else constant.
A key property of Double ML is its "double robustness" - the estimator remains consistent and asymptotically normal even if one of the first-stage models is misspecified, as long as the other model is correctly specified. As chernozhukov2018double note, "Both components need to be explicitly estimated to construct a debiased estimator. Using both components, however, yields an extra degree of robustness" (p. C5). This robustness property is particularly valuable for measuring supply incrementality because it helps separate the true causal effect from correlation due to common demand drivers, even in cases where either the supply or bookings model might be imperfectly specified.
For model selection for the first-stage models, we evaluated various algorithms based on validation data performance. Among the considered models, \href{http://lightgbm.readthedocs.io/}{LightGBM} demonstrated better out-of-sample fit for both predicting outcome $Y$ and treatment $S$, making it our preferred choice for the first-stage predictions in the Double ML framework.
For model estimation and validation, we employ a time series split approach rather than traditional random cross-validation to avoid overfitting due to shared seasonality across listing segments. This choice is crucial when working with time series data like ours, where random splitting would violate the temporal structure and lead to data leakage problems. Time series data exhibits temporal dependencies that make observations from nearby time periods correlated, and random splitting would allow models to "peek into the future" during training, resulting in overly optimistic performance estimates (bergmeir2018note).
As mentioned above, while we can abstract away from substitution between the listing groups ($g$), we need to account for potential substitution between listing segments ($j$). Therefore, for predicting $Y$ and $S$ in the first stage for listing segment $j$, besides the covariates $X$ in listing segment $j$ ($X_{g,j,t}$), we also account for the covariates $X$ in other listing segments $j$ within the same group $g$ ($X_{g,-j,t}$), as well as available supply in those other listing segments ($S_{g,-j,t}$).
Rather than treating all listing segments independently, we leverage these relationships through a geospatial approach that incorporates similarity measures between listing segments (credit2024structured). The geospatial approach leverages the insight that listing segments that are most similar to each other are likely to have similar patterns of demand and supply dynamics due to underlying spatial and feature-based dependencies. Incorporating measures of similarity between listing segments results in improved predictions in the first stage model and reduced variance in the second stage treatment effect estimation.
In the second stage, we parameterize the heterogenous treatment effect as a function of $X$. We adopt a Bayesian approach to leverage this pre-existing knowledge (sanchez2025airbnb) as priors and only update our belief where new data signals strongly suggest deviations from the prior.
Specifically, we estimate a hierarchical model as follows, formulating the heterogenous treatment effect as a combination of listing group level random effects and the interaction effects reflecting how supply incrementality varies with $X$ in a listing segment as well as the most similar listing segments in the same group.
In this specification, $\alpha_g \sim N(E_g, \sigma_\alpha^2)$ is a random effect reflecting listing group level base supply incrementality, using listing group level supply incrementality $E_g$ from the previous model to inform prior distributions. The other two terms reflect the interaction effects with the covariates in the listing segment as well as in its nearest neighbors, with the parameters following uninformative priors $\lambda \sim N(0, \sigma_\lambda^2)$ and $\tilde{\lambda} \sim N(0, \sigma_{\tilde{\lambda}}^2)$.
Our findings demonstrate that the combined approach of causal machine learning with geospatial relationships and hierarchical Bayesian modeling provides plausible and robust estimates of supply incrementality across heterogeneous listing segments in the Airbnb marketplace. The methodology successfully captures the varying impact of additional supply across different listing segments, identifying segments where additional supply would bring the most value to the community.
Figure (ref) illustrates the prior and posterior estimates for the most important interaction coefficients ($\lambda$) that determine heterogeneous incrementality estimates. We find that most of the posterior estimates are consistent with the prior, but more precise (a narrower 95% credible interval compared to the 95% confidence interval of the prior). Further, where the estimated posterior changes signs relative to the prior, the change makes sense.
Figure (ref) compares the estimated supply incrementality (aggregated to the product group level, weighted by supply) vs. the prior estimate. In this figure, the size of the circle reflects the average supply level of the product group and the color of the circle reflects historical demand levels in the listing group.
This figure shows that in the largest groups where the posterior estimates differ significantly from the prior, the difference can often be explained by demand levels, suggesting that our model is capturing meaningful marketplace dynamics when substantial data is available to overcome the prior.
Findings at the listing segment level are also corroborated with other research and domain knowledge. The pattern of listing segment-level incrementality aligns well with our understanding of marketplace dynamics, showing higher incrementality in areas with strong demand signals and unique listing attributes that are not easily substitutable. This validation through domain expertise gives us confidence in the model's ability to capture meaningful variations in the causal impact of supply across different market segments. The posterior estimates at the listing segment level provide scalable, consistent insights about supply incrementality that enable important business decisions to bring the most value to the community.
Model evaluation in our setting is particularly challenging due to the fundamental problem of causal inference - we never observe the counterfactual outcomes (sanchez2025airbnb;gelman2006data). The ideal evaluation would compare our estimates to results from randomized experiments that randomize supply directly, but such experiments are typically infeasible in two-sided marketplaces. Instead, we focus on validating our model through consistency with theoretical expectations, alignment with domain expertise, and robustness to alternative specifications.
We validate the model output using the following approaches:
This paper introduces a novel approach to estimating supply incrementality in two-sided marketplaces with heterogeneous products by combining causal machine learning techniques with geospatial methods and Bayesian hierarchical modeling. Our methodology addresses several critical challenges in measuring the causal impact of additional supply on marketplace outcomes, particularly the endogeneity between supply and demand, substitution effects between similar products, and the need to account for pre-existing knowledge.
Beyond its application to Airbnb's marketplace, our methodology has broad relevance for other two-sided platforms facing similar challenges in understanding the incrementality of additional supply or products. For example, this approach could be adapted to ride-sharing platforms evaluating the impact of additional drivers, e-commerce marketplaces considering new sellers, or content platforms assessing additional creators.
Future research could extend this framework by incorporating temporal dynamics, exploring alternative methods for defining similarity between product segments, and developing experimental approaches to validate the causal estimates. As marketplace platforms continue to grow and diversify their offerings, methodologies that provide granular, causal insights into marketplace dynamics will become increasingly valuable for business decisions to bring more value to the community.
The authors would like to thank Brian Weller (Airbnb, Inc.), Mitra Akhtari (Airbnb, Inc.), Totte Harinen (Airbnb, Inc.), Thomas Covert (Airbnb, Inc.), Carlos Sanchez-Martinez (Airbnb, Inc.), Yvonne Wang (Airbnb, Inc.), Allen Ross (Airbnb, Inc.), Shanni Weilert (Airbnb, Inc.), Navin Sivanandam (Airbnb, Inc.) for their contributions and support for this project.
\nocite{berry1995auto}