Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
99,979 characters · 14 sections · 120 citation commands
Estimating Demand with Recentered Instruments
\setcounter{page}{1} {6pt plus 3pt minus 2pt} {6pt plus 3pt minus 2pt}
Many economic analyses depend on accurate estimates of the demand for differentiated products. Prominent examples from the field of industrial organization (IO) include measuring welfare effects of mergers or new products and testing models of firm conduct; trade economists similarly use demand estimates to measure welfare effects of new tariffs and gains from trade or internal migration. Often, these analyses leverage structural models of demand that allow for rich and realistic substitution patterns—such as the mixed multinomial logit model popularized by Berry1995. To estimate these models with market-level data, researchers need multiple instrumental variables (IVs) that address the endogeneity of prices and other terms capturing the substitution patterns.
This paper develops a new approach to constructing powerful instruments for popular demand models by leveraging a set of exogenous supply-side shocks (for brevity, “cost shocks”) like input price changes, new taxes or subsidies, markup regulations, or certain productivity and ownership shocks. Such shocks are increasingly found in empirical applications, where they serve as a natural instrument for price.\footnote{ Examples include Berry1999 and Goldberg2001 (exchange rate shocks), Market2016 (subsidies), miller_understanding_2017 (merger shocks), and nakamura_accounting_2010 (productivity shocks). } However, even when plausibly exogenous cost shocks are available, researchers typically identify the parameters governing substitution patterns (so-called “nonlinear” parameters) using other IVs constructed from observed characteristics of competing products. Prominent examples include nest size instruments in nested logit models, “BLP instruments” which average or sum the characteristics of a product's competitors, and more sophisticated versions like the efficient IVs of Berry1999 and the differentiation IVs of Gandhi2015. Such instruments rely on the econometric exogeneity of product characteristics: a strong assumption that is often inconsistent with natural models of product entry ackerberg_estimating_2009,petrin_identification_2022. Characteristic-based IVs can also be weak, and lack across-market variation when all markets have the same products Reynaert2014,Nevo2001.\footnote{ BLP instruments are sometimes also used to estimate the price sensitivity parameter, when cost shocks are not available. Armstrong2016 studies the weak instrument problem that arises in that context. }
We propose instruments that combine product characteristics and cost shocks in a particular way: to predict the response of the demand model's endogenous variables to the exogenous shocks. These IVs are motivated by thinking of the model as structuring “spillover effects,” of exogenous changes to product prices on the market shares of other products. For example, the nested logit model structures spillovers with a parameter $\sigma$ that governs whether, when prices exogenously rise, consumers substitute primarily to local competitors in a product's nest or more evenly to all unaffected products in the market. To distinguish between those cases and thus identify $\sigma$, we propose instruments which predict how a product's within-nest market share changes in response to a set of cost shocks. A simple IV in this spirit is the the cost shock of a product less the average shock in its nest. We show how such IVs can be generally constructed from first-order approximations to model-implied responses, yielding shift-share instruments with cost shocks as the exogenous “shifts.” We also propose a novel instrument construction from exact model-based predictions. We build intuition for these constructions in mixed logit models by considering instruments constructed to predict the impact of shocks via small “nonlinear” parameters (i.e., in a “local to logit” approximation, similar to salanie_fast_2022).
Instruments constructed this way are generally complex formulas of the exogenous cost shocks and the likely endogenous product characteristics. To avoid bias from the latter, we follow BH1 in recentering the IVs: i.e., subtracting their expectation over the data-generating process of exogenous shocks. For example, a researcher may simulate this process by permuting observed shocks across comparable products; she could then recenter any formula IV by subtracting from each product's instrument value the average value across these counterfactual shocks, holding the characteristics of all products fixed. When the instruments are constructed from first-order approximations (i.e. as shift-share IVs) recentering is simpler as it only requires specifying and adjusting for the conditional mean of the shocks. In general, recentering ensures our IVs derive their validity only from the exogeneity of cost shocks—even though they also derive power from product characteristics.
We formalize this approach in a broad class of demand models, which includes both conventional mixed and nested logit from IO as well as analogous constant elasticity of substitution (CES) models from trade. We focus on estimating the parameters that govern own- and cross-price elasticities, which are central to many important policy counterfactuals. Consistency of recentered IV estimates follows when there are either many uncorrelated markets or many uncorrelated shocks that can affect multiple markets jointly while inducing sufficient across-product variation in the instruments. Unlike conventional characteristic-based IVs, our instruments can yield consistent estimates when all markets have the same products and product fixed effects are included. Asymptotic normality of our estimators follows with many market clusters; we further extend results in adao2019shift and BJH2018 to show how many-shock asymptotic inference can be conducted with shift-share instruments. We characterize the asymptotically efficient recentered IVs, building on chamberlain1987asymptotic,chamberlain1992efficiency, newey1994large, and borusyak_efficient_2021. While our baseline analysis shows how powerful recentered IVs can be derived for a given demand model, we also consider non-parametric identification by building on Erry2014. Notably, our approach applies even when the same products are sold in all markets and product fixed effects are included, as in Nevo2001; characteristic-based IVs have no variation in those settings.
We compare this approach to conventional ones in a series of Monte Carlo simulations based on the data-generating process in Gandhi2015. When characteristics are exogenous, the power of recentered instruments—whether derived from first-order approximations or exact model-based predictions—is comparable to that of Differentiation IVs and much better than that of BLP instruments. Expectedly, recentered IVs have less power with a lower variance of cost shocks while the power of characteristics-based IVs is lower with less variation in choice sets across markets. A simple model of strategic product entry introduces significant bias in characteristic-based IV estimates, while recentered IV estimates remain accurate.
This paper contributes to two main literatures. First, we contribute to an IO literature studying demand estimation without exogenous characteristics. The potential bias from endogenous characteristics has been noted as far back as Berry1995. Existing solutions to this concern broadly fall into two categories: some papers put additional structure on the model unobservables (i.e., the unobserved taste shifters) by assuming characteristic endogeneity is captured by controls (e.g., product fixed effects in Nevo2001) or imposing a particular statistical process for the unobservables (e.g., sweeting2013dynamic and moon2018estimation). Other papers explicitly models characteristic choice or product entry (e.g., crawford2019quality and petrin_identification_2022).\footnote{ See also fan2013ownership, who builds BLP-type instruments for a firm's endogenous characteristics from particular characteristics of the firm's competitors: namely, consumer characteristics in markets where competitors operate. } In contrast, our approach places no restrictions on how the model unobservables relate to observed characteristics and does not require a model of entry, relying only on cost shock exogeneity. This solution relates to an idea in ackerberg_estimating_2009 of searching for “orthogonal instruments” to identify own- and cross-price elasticities while leaving the relationship between characteristics and taste shifters unidentified. We propose a concrete way to achieve this goal, via recentered functions of exogenous cost shocks and endogenous characteristics.\footnote{ A larger literature improves mixed logit demand estimation in other ways, maintaining the assumption of exogenous characteristics. See, e.g., Berry1999, Reynaert2014, and Gandhi2015 on IV power, salanie_fast_2022 and lu_semi-nonparametric_2023 on alternative estimation methods, and wang_sieve_2023 on allowing for non-parametric distributions of random coefficients. }
Second, we contribute to a recent econometrics literature on identification and estimation with shift-share IVs and other “formula” instruments combining exogenous shocks with other potentially endogenous data (BJH2018,borusyak_design-based_2023; adao2019shift; BH1,borusyak_efficient_2021). While this literature studies linear causal or structural models, we focus on nonlinear demand estimation. In this sense our work is also related to Adao2018a who identify parameters of a quantitative spatial model by the responses of endogenous variables to exogenous shocks; borusyak2022understanding follow a similar approach with a migration model. Notably, both of these papers work with linear approximations of their models—introducing inaccuracies when shocks are large—while we work directly with nonlinear demand models.\footnote{ adao2024putting develop a similar approach to specification testing that applies to nonlinear models, but they do not propose an estimation procedure. }
Recent complementary work by Andrews2022 shows that recentered instruments are more robust to demand model misspecification than characteristic-based IVs, in the sense of recovering more interpretable causal estimands in a non-parametric potential outcomes framework. We demonstrate a different advantage of recentered IVs: that they can be used to relax the assumption of exogenous characteristics in a given demand specification. Moreover, we propose specific constructions of powerful recentered IVs tailored to a class of demand models.
Finally, our analysis relates to empirical studies using weighted sums or other transformations of shocks as instruments to estimate demand models. Recent examples include costinot2016evolving, Adao2017, couture2020urban, fajgelbaum2020return, adao2022imports, barahona2023equilibrium, and adao2024putting; see also fujiy_production_2024 who estimate input demand by firms. Typically these instrument constructions arise from intuitive arguments instead of being derived as model-implied responses to the shocks, limiting their power. Moreover, while the constructions are sometimes simple enough to not need recentering, this consideration is also not typically part of the formal analysis. We show in a general setting how powerful instruments can be constructed by leveraging the structure of the model, and how instrument validity may be made more credible and transparent via explicit recentering.
The rest of this paper is structured as follows. Section (ref) builds intuition for our approach in a simple nested logit demand model with randomized cost shocks. Section (ref) develops our general approach and discusses asymptotic properties. Section (ref) illustrates the approach with simulations. Section (ref) concludes. All proofs are collected in the appendix.
We start with a simple example that illustrates the main logic of our approach as well as its advantages over conventional methods. Here we keep the presentation informal and intuitive, leaving formal assumptions and results for the more general analysis in Section (ref).
Consider a set of markets $m$, each with a set of differentiated products $j\in\mathcal{J}_{m}$ and an outside good $j=0$. The products are grouped into “nests” $n(j)$; let $d_{jn}=\mathbf{1}\left[n(j)=n\right]$ denote mutually exclusive nest indicators. A researcher observes the nest allocation, along with the price $p_{jm}$ and quantity share $s_{jm}$ of each product. Finally, the researcher observes a set of shocks $g_{jm}$ which increase products' marginal costs (e.g., via input prices) but do not directly affect demand.
The researcher correctly assumes that market shares arise from a nested logit demand model: a mass of consumers $i$ in each market choose a single product or the outside good to maximize their utility $u_{ijm}=\alpha p_{jm}+\xi_{jm}+\varepsilon_{ijm}$, where $\xi_{jm}$ is a common taste shifter and $\varepsilon_{ijm}$ is an idiosyncratic taste shock that can be correlated across products in a nest. The outside good has a zero price and taste shifter, such that utility from it is $u_{i0m}=\varepsilon_{i0m}$. Conditional on the prices and taste shifters, the idiosyncratic shocks $\left(\varepsilon_{ijm}\right)_{j\in\mathcal{J}_{m}\cup\left\{ 0\right\}}$ are distributed across consumers in such a way that the market shares satisfy:
where $s_{0m}$ is the market share of the outside good in market $m$ and $s_{nm}$ is the total market share of products in nest $n$ and market $m$.\footnote{ Appendix (ref) gives the formula for nested logit market shares and derives this expression from it. The nested CES model, commonly used in international trade and spatial economics, implies similar expressions; see Section (ref). } From this equation the researcher is interested in estimating $\alpha<0$, which determines the own-price sensitivity of demand, and $\sigma\in[0,1)$ which captures the extent of within-nest correlation in the taste shocks that governs substitution patterns. These two parameters determine the matrix of cross-price elasticities, which is a key input to a variety of policy counterfactuals (e.g., merger analyses) and consumer welfare calculations.
Estimating $\alpha$ and $\sigma$ generally requires finding two instruments which are uncorrelated with the unobserved taste shifters $\xi_{jm}$ but correlated with the two endogenous variables in equation ((ref)): price $p_{jm}$ and the log within-nest market share $\log(s_{jm}/s_{n(j)m})$. Here price endogeneity likely arises because more popular products (with higher $\xi_{jt}$) are likely of higher quality and therefore more expensive to produce; firms may moreover optimally charge higher markups for them. Endogeneity of $\log(s_{jm}/s_{n(j)m})$ further arises from the market shares' direct dependence on the taste shifters: i.e., more popular products will have larger within-nest market shares.
A natural instrument for the own-price sensitivity parameter $\alpha$ is the excluded cost shock $g_{jm}$, since higher costs are predicted to at least partially pass through to higher prices.\footnote{ We attribute an instrument to a particular endogenous variable informally; technically both instruments jointly identify both parameters when they are valid. } To justify this choice simply, suppose the $g_{jm}$ are drawn in a randomized trial after product entry. Randomization and the natural exclusion restriction that the cost shocks do not directly affect demand ensure that $g_{jm}$ is a valid instrument for equation ((ref)), i.e. that $\cov{g_{jm},\xi_{jm}}=0$.\footnote{ Throughout, we call instruments “valid” when they are uncorrelated with the model error (i.e., the unobserved taste shifter). We consider an instrument's relevance, i.e. its correlation with endogenous variables, separately. }
It is more difficult to find an instrument for the substitution parameter $\sigma$. One popular strategy is to construct instruments from the observed characteristics of other products, such as the nest indicators $d_{jn}$. In particular, in nested logit models, it is common to use the number of products in $j$'s nest, $N_{n(j)m}=\sum_{k\in\mathcal{J}_{m}}d_{kn(j)}$ (or, equivalently, the number of other products excluding $j$; e.g., Goldberg2001, town2003welfare, miller_understanding_2017). This instrument is expected to predict $\log\left(s_{jm}/s_{n(j)m}\right)$ because a product with a larger number of “local” competitors in its nest should have, on average, a smaller within-nest share.
Such characteristic-based instruments have at least two drawbacks. First, their validity hinges on the econometric exogeneity of the characteristics: a strong assumption that can be at odds with natural models of product entry. For example, the nest size instrument will be invalid (with $\cov{N_{n(j)m},\xi_{jm}}>0$) when firms introduce more products in nests for which consumers have a higher preference in a particular market—a natural tendency of profit-optimizing firms with at least partial information on consumer tastes (see, e.g., aguiar2018quality). Even if the instrument is based on the product entry by firms other than the producer of $j$, it is not econometrically exogenous. Second, such instruments have no useful variation in some common empirical contexts. Specifically, $N_{n(j)m}$ and similar instruments cannot be used if all products are sold in all markets and product fixed effects are included (as is commonly done since Nevo2001). Given these issues, we look for different instruments.
Consider an IV that measures the relative cost shock of product $j$ vs. the average shock in its nest:
We next explain why $z_{jm}$ is both relevant and valid, and how it exemplifies our general approach to constructing instruments as certain combinations of exogenous cost shocks and potentially endogenous characteristics (here, nest indicators) of all the products in the market.
First, this instrument is likely relevant (i.e., correlated with $\log\left(s_{jm}/s_{n(j)m}\right)$): if product $j$'s local competitors in its nest have relatively low cost shocks, its within-nest market share is expected to be lower. This intuitive argument has a formal backing: $z_{jm}$ approximates the model-predicted response of the endogenous variable $\log\left(s_{jm}/s_{n(j)m}\right)$ to the exogenous shocks. Specifically, consider a hypothetical scenario in which all products have equal prices $\check{p}_{km}=\check{\pi}_{0}$ and unobserved taste shifters $\check{\xi}_{km}=0$, and thus all products within the nest have the same market shares. To this scenario we introduce an exogenous component of price variation, $\hat{p}_{km}=\check{p}_{km}+\check{\pi}g_{km}$ for some auxiliary constant $\check{\pi}\ne0$ that aims to capture the pass-through of cost shocks into prices in a simple way. Nested logit demand characterizes the share response to this set of price changes; Appendix (ref) shows that, in a first-order approximation that is precise for small shocks, the resulting change in $\log\left(s_{jm}/s_{n(j)m}\right)$ is equal to $z_{jm}$, up to a constant scaling factor. While cost shocks is only one of the reasons why $\log\left(s_{jm}/s_{n(j)m}\right)$ deviates from equal shares, $z_{jm}$ captures the component of variation in the within-nest shares due to the cost shocks and is thus likely relevant.
Second, $z_{jm}$ is a valid instrument (i.e., $\expec{z_{jm}\xi_{jm}}=0$) when the cost shocks are exogenous in the sense we previously assumed to justify their use as an instrument for $\alpha$. This claim is nontrivial because the formula for $z_{jm}$ incorporates not only the cost shocks but also the nest dummies, which are likely econometrically endogenous. Nevertheless, $z_{jm}$ is constructed in such a way that its validity stems only from the exogeneity of the shocks only. Intuitively, when cost shocks are random, it is also random whether the shock for a particular product is higher or lower than the average in its nest. Formally, $z_{jm}$ is a recentered instrument, meaning its expectation over draws of the shocks is zero conditional on the other variation BH1: $ \expec{z_{jm}\mid\left(d_{kn}\right)_{k\in\mathcal{J}_{m},n}} = \expec{g_{jm}} - \frac{1}{N_{n(j)m}}\sum_{k\in\mathcal{J}_{m}}d_{kn(j)}\expec{g_{km}} = 0 $ when cost shocks are drawn randomly. Thus, $z_{jm}$ is guaranteed to be uncorrelated with any function of the nest allocation which could create endogeneity problems. Other formula instruments can be adjusted via a recentering procedure that removes the component correlated with the characteristics—as we soon illustrate.
To recap, our proposal is to construct instruments as recentered model-based predictions of how relevant endogenous variables respond to the cost shocks. This general approach suggests three ways of improving the simple $z_{jm}$ instrument, in the sense of likely power gains, by forming better predictions of the endogenous variable. First, consider an exact model-based prediction of $\log\left(s_{jm}/s_{n(j)m}\right)$ from the price predictions $\left(\hat{p}_{km}\right)_{k\in\mathcal{J}_{m}}$ in place of the first-order approximation in $z_{jm}$. Appendix (ref) shows this prediction can be written:
where $\check{\alpha}$ and $\check{\sigma}$ are some preliminary estimates of $\alpha$ and $\sigma$ and $\check{\pi}$, as before, is an estimate of the pass-through of cost shocks into prices.\footnote{ Preliminary estimates $\check{\alpha},\check{\sigma}$ can be obtained using simpler instruments, such as ((ref)), in a first step, while $\check{\pi}$ can be obtained from a regression of prices and the own-product shocks. In Section (ref) we show how a continuously updating estimator can bypass the need for $\check{\alpha},\check{\sigma}$. } Unlike $z_{jm}$, exogeneity of the cost shocks does not make this prediction a valid instrument; instead, like the conventional nest size instrument $N_{n(j)m}$, its validity also hinges on the econometric exogeneity of the nest allocation. To see this simply, note that $ \widehat{\log}\left(s_{jm}/s_{n(j)m}\right) $ varies over products even in the absence of cost shocks ($g_{km}=\mu_{g}$ for all $k$). In fact, this variation is driven exactly by nest size: plugging in $g_{km}=\mu_{g}$ for all $k$ yields $ \widehat{\log}\left(s_{jm}/s_{n(j)m}\right)=-\log N_{n(j)m} $, showing that this prediction suffers from exactly the same endogeneity concerns as the conventional nest size instrument.
Following BH1, we propose obtaining valid instruments by recentering model-based predictions like ((ref)), using knowledge of how the exogenous shocks are drawn. Recentering is straightforward when the shocks are drawn randomly in an experiment: the researcher can re-draw many sets of counterfactual shocks from the experimental protocol, recompute the prediction under each set, average across shock counterfactuals to measure the expected prediction ($ \mu_{jm} \equiv \expec{ \widehat{\log}\left(s_{jm}/s_{n(j)m}\right) \mid \left(d_{kn}\right)_{k\in\mathcal{J}_{m},n} } $ in the case of ((ref))), and subtract this expectation from the actual prediction.\footnote{ We note that $\mu_{jm}$ is related but not equal to the value of $z_{jm}$ with no shocks, $-\log N_{n(j)m}$. The difference arises because $z_{jm}$ is a nonlinear function of the shocks, and thus taking the expectation of $z_{jm}$ across the shock distribution is not the same as plugging in expectation of the shocks. } Like $z_{jm}$, the recentered prediction
is a valid instrument regardless of any econometric endogeneity of the nest allocation. Below we discuss other ways to recenter predictions in observational data, where the shock data-generating process is unknown. Although recentering removes some variation in the prediction, which was not necessary with $z_{jm}$, starting from a better prediction still improves the first-stage borusyak_efficient_2021. Note that inaccuracy of the initial $(\check{\alpha},\check{\sigma})$ estimates in equation ((ref)) is not an issue for the validity of $z_{jm}^{\text{exact}}$, given recentering, though it will likely affect power.
A second type of improvement comes from using additional data to better predict the endogenous variables' responses to exogenous shocks. One particularly useful input is the market shares $s_{jm}^{\text{pre}}$ and prices $p_{jm}^{\text{pre}}$ for the same products and the same market in an earlier period, before the shocks $g_{km}$ were drawn. To the extent that prices and taste shifters are serially correlated, this yields better predictions of prices and market shares in the period of interest.\footnote{ A second complementary use of such data is that equation ((ref)) can be estimated in time-differences, yielding more precise estimates from the same instruments when $\xi_{jm}$ is serially correlated. We discuss this approach in Section (ref). } The instrument that results from incorporating this information is also very intuitive: Appendix (ref) shows that the first-order approximation of the model-predicted response of $\log\left(s_{jm}/s_{n(j)m}\right)$ to the exogenous shocks around the pre-period shares (rather than equal shares) yields
Like $z_{jm}$, this prediction is mean-zero over draws of random cost shocks, $ \expec{z_{jm}^{\text{weighted}}\mid\left(d_{kn}\right)_{k\in\mathcal{J}_{m},n}}=0 $, making it a recentered instrument without further adjustment.\footnote{ $z_{jm}$ and $z_{jm}^{\text{weighted}}$ are examples of shift-share instruments, which average the exogenous shocks with a set of weights capturing differential shock exposure BJH2018. As discussed below, such instruments are often easier to recenter or require no recentering at all because they are linear in the shocks. } The appendix further shows that the two improvements can be combined: a researcher can incorporate the lagged share information to improve the recentered exact prediction ((ref)), too. In either case, the recentered instrument will again be valid just by virtue of the exogenous shocks—even though it now draws power from variation in lagged market shares (as well as the nest allocation) and lagged market shares by themselves need not be exogenous.\footnote{ We note that, unlike the nest size IV, $z_{jm}$ and $z_{jm}^{\text{weighted}}$ have cross-market variation even if all products are sold in all markets—so long as cost shocks vary across markets. Moreover, $z_{jm}^{\text{weighted}}$ can vary across markets even if cost shocks do not, if the pre-period shares vary due to any unobserved cost or taste differences. }
Finally, consider an instrument which uses a more realistic prediction of how prices respond to the full set of exogenous shocks. A researcher might, for example, specify an auxiliary pricing model which captures not only the pass-through of product $j$'s cost shock to its own price but also how $p_{jm}$ responds to competitor cost shocks $g_{km}$ for $k\neq j$ (depending, for instance, on whether $j$ and $k$ are offered by the same firm). Substituting this model's price predictions into any of the above instrument constructions, in place of the simple $\hat{p}_{jm}=\check{\pi}g_{jm}$ prediction, yields an instrument which is likely more powerful when such cost shock spillovers are important and can be estimated. The better price prediction can also be recentered and used in place of $g_{jm}$ as an instrument for identifying $\alpha$. Note that as with the initial $(\check{\alpha},\check{\sigma})$ estimates in $z_{jm}^{\text{exact}}$, the validity of these instruments does not hinge on the accuracy of the pricing model.
We now consider a broader class of demand models and formalize our general approach. Section (ref) develops the baseline mixed logit model and the introduces key shock exogeneity assumption. Section (ref) defines recentered IVs and establishes identification with them. Section (ref) develops our proposal for constructing powerful recentered IVs from the structure of the model, while Section (ref) establishes consistency and asymptotic inference. Section (ref) discusses several extensions to the baseline model.
We consider a class of random utility models—canonical mixed logit demand—with market-level data as in Berry1995; see Berry2021 and Gandhi2021 for more recent treatments. A researcher observes a set of markets $m$ (which might correspond to regions, periods, or both) with differentiated products $j\in\mathcal{J}_{m}$, prices $p_{jm}$, and quantity shares $s_{jm}$.\footnote{ Here we do not restrict whether the data consist of many markets or just a single one, whether the markets are randomly sampled, or whether the number of products per market is large. We return to these issues in Section (ref). } Each product also has a vector of observed characteristics $x_{jm}\in\mathbb{R}^{L}$ (which includes an intercept), as well as an unobserved scalar taste shifter $\xi_{jm}$. All variables are normalized such that the outside good in each market, $j=0$, has $p_{0m}=\xi_{0m}=0$ and $x_{0m}=0$.
Consumers $i$ choose among all products and the outside good by maximizing their utility:
Here product $j$'s mean utility $\delta_{jm}$ is determined by its price, characteristics, and the taste shifter:
with $\alpha<0$ and $\beta\in\mathbb{R}^{L}$. A subvector of characteristics $x_{jm}^{(1)}=\left(x_{jm1},\dots,x_{jmL_{1}}\right)$, as well as (potentially) price, also enter utility with mean-zero “random coefficients” $\eta_{i}=\left(\eta_{i\ell}\right)_{\ell=0}^{L_{1}}$ that capture heterogeneous consumer preferences. This $\eta_{i}$ is iid across consumers and follows distribution $\mathcal{P}\left(\cdot;\sigma\right)$ that is known up to a vector of “nonlinear” parameters $\sigma$ (typically Gaussian with independent components and standard deviations $\sigma_{0},\dots,\sigma_{L_{1}}$). Finally, $\varepsilon_{ijm}$ is an extreme-value shock, iid across consumers and products including the outside good. Integrating out these shocks implies market shares satisfy
where bold symbols denote a collection of variables for all products in the market: $\boldsymbol{v}_{m}=\left(v_{jm}\right)_{j\in\mathcal{J}_{m}}$ for any variable $v_{jm}$. The distribution $\mathcal{P}$ determines the patterns of product substitutability: for instance, the pure multinomial logit model corresponds to no variation in random coefficients (typically captured by $\sigma=0$). The nested logit model considered in Section (ref) is also a special case, with nest dummies as characteristics and with a particular choice of $\mathcal{\mathcal{P}}(\cdot;\sigma)$ mcfadden_modelling_1978. berry_estimating_1994 and Berry1995 famously show that the share function $ \mathcal{\mathcal{S}}_{j}(\boldsymbol{\delta}_{m};\sigma,\boldsymbol{x}_{m}^{(1)},\boldsymbol{p}_{m}) $ is invertible, such that mean utilities can be derived from $\sigma$ and observed data:
for functions $\mathcal{D}_{j}(\cdot)$ that generally do not have a closed-form but can be computed numerically.
We focus on estimating parameters $\theta=(\alpha,\sigma')'$, which are central to a number of important policy counterfactuals that do not involve changing product characteristics. Indeed, the model implies that own- and cross-price elasticities can be characterized in terms of $\theta$ and observed data:
where \[ s_{ji} = \frac{ \exp\left( \mathcal{D}_{j}\left( \boldsymbol{s}_{m};\sigma,\boldsymbol{x}_{m}^{(1)},\boldsymbol{p}_{m} \right) + \eta_{0i}p_{jm} + \sum_{\ell=1}^{L_{1}}\eta_{i\ell}x_{jm\ell} \right) }{ 1 + \sum_{k\in\mathcal{J}_{m}} \exp\left( \mathcal{D}_{k}\left(\boldsymbol{s}_{m};\sigma,\boldsymbol{x}_{m}^{(1)},\boldsymbol{p}_{m}\right) + \eta_{0i}p_{km} + \sum_{\ell=1}^{L_{1}}\eta_{i\ell}x_{km\ell} \right) }. \] In contrast, a consistent estimate of the causal effect of characteristics on mean utility, $\beta$, is not needed when analyzing a merger (ackerberg_estimating_2009). Similarly, the welfare gains from a new product can be computed without $\beta$.\footnote{ While such gains require a prediction of the new product's mean utility, naturally based on its characteristics, this constitutes a prediction problem and not a causal problem which would require the structural parameter $\beta$. We are not aware of previous work making this point. }
To identify $\theta$ we leverage share inversion, which implies a structural equation that is additive in the unobserved taste shifter: \[ \mathcal{D}_{j}\left(\boldsymbol{s}_{m};\sigma,\boldsymbol{x}_{m}^{(1)},\boldsymbol{p}_{m}\right) = \alpha p_{jm} + \beta'x_{jm} + \xi_{jm}. \] This representation permits estimation of $\theta$ from moment conditions of the form
for some vector of instruments $Z_{jm}$ that are uncorrelated with $\xi_{jm}$. If the instruments are also uncorrelated with $x_{jm}$ (as ours will be), equation ((ref)) will hold for any value of $\beta$.
To build instruments, we assume the researcher observes a set of supply-side shocks $g_{jm}$ that vary by product and market. Shocks to input prices is a natural source of supply-side shocks commonly used in the IO literature (e.g., Villas-Boas2007, backus_common_2021, barahona2023equilibrium); other studies have used exchange rate shocks (e.g., Berry1999,Goldberg2001), shocks from weather (as a productivity shifter; e.g., nakamura_accounting_2010), product-specific subsidies Market2016, taxes Dearing2022, and markup shocks due to mergers miller_understanding_2017.\footnote{ We do not require the shocks to be independent across products and markets. For instance, all cars produced in the same country are assigned the same exchange rate shock, and input price shocks affect all products using this input, to different extents. For now we assume that the researcher can assign each product to the corresponding country of production or shares of different inputs but we relax this assumption in Section (ref). } We generically call the $g_{jm}$ “cost shocks” and assume that they are exogenous in the following sense:
That is, we assume the taste shifter of each product $j$ is mean-independent of cost shocks (both for $j$ and its competitors), conditionally on the observed characteristics of all products.
To understand the economic content of the assumption, note that the unobserved taste shifter captures both objective characteristics of the product chosen by firms and the subjective preferences of consumers. Thus, several conditions have to be met for equation ((ref)) to hold. First, cost shocks should not affect product entry decisions or firm choices of unobserved characteristics. A sufficient condition is that cost shocks are realized after those decisions have been made (but before prices are set—a condition required for relevance of cost shocks as instruments). Second, cost shocks should not influence consumer preferences. Such influence could be possible when cost shocks influence firms' advertising decisions or stem from the prices of inputs that affect consumer earnings.
In addition to these exclusion restrictions, cost shocks may not be correlated with variables that affect product entry or consumer preferences. This can be viewed as an independence condition, in the sense of imbens1994identification, that is automatically satisfied in randomized experiments. Examples from the literature show how it can also hold in observational data. Several IO papers use the exchange rate in the country of production as a cost shock for automobiles Berry1999,Goldberg2001,Grieco2021 and other industries nakamura_accounting_2010. While these studies use the level of the exchange rate, their changes over time may be particularly attractive as the $g_{jm}$ because exchange rates are known to roughly follow a random walk (e.g., kilian2003so). The same argument applies to many commodity price changes, for goods using these commodities as inputs (e.g., coffee in nakamura_accounting_2010). For inputs traded on futures markets, ackerberg_estimating_2009 point out that the difference between the realized input price at the time when downstream firms set prices and the price of a futures contract at the earlier moment when product entry has been decided is guaranteed to be unrelated to the characteristics.
While the assumption in ((ref)) is restrictive, it is important to highlight what it does not entail: it allows unobserved taste shifters $\xi_{jm}$ to be arbitrarily correlated with the observed characteristics of both product $j$ and its competitors.\footnote{ An additional feature of ((ref)) is that it allows the cost shocks to be mutually correlated. Mutual correlations of taste shifters are also allowed, as when unobserved market-level demand and cost conditions affect the choice of unobserved quality of all firms in the market. } This is in contrast to the prevalent approach in the literature Berry1995,Berry1999,Gandhi2015 which imposes a stronger assumption:
equivalent to imposing $\expec{\xi_{jm}\mid\boldsymbol{x}_{m}}=0$ in addition to equation ((ref)). Under this stronger condition, instruments constructed as functions of own and competing products' characteristics are valid. This includes BLP instruments, computed as sums or averages of competitor characteristics, efficient instruments from Berry1999, as well as the differentiation IVs of Gandhi2015 which capture the average distance between $x_{jm}$ and characteristics of competitors or the number of products in the market with characteristics sufficiently close to $x_{jm}$.
As recognized as far back as Berry1995, however, the econometric exogeneity of observed characteristics is an unappealing restriction on product entry. It is often arbitrary which objective characteristics are observed or unobserved by the econometrician, such that there is no reason why the two groups should be uncorrelated with each other.\footnote{ Note, however, that it is generally necessary for characteristics with random coefficients to be observed. An interesting exception is provided by Adao2017 who show identification in a model with a random coefficient on unobserved mean utility; our approach applies to that model as well. } Moreover, in natural models of product entry, $\xi_{jm}$ can be related to characteristics of competing products. This can happen, for instance, when all firms observe some information about the cost or demand conditions common to the market when making product entry decisions. Similar to the discussion in Section (ref), in a market where consumers like small and fuel-efficient cars (i.e. their $\xi_{jm}$ is predicted to be higher), we expect all firms to pivot towards these characteristics, whether in ways observed or unobserved to the econometrician. In this situation, BLP and differentiation IVs need not be valid.
In the rest of our analysis we maintain a modified version of ((ref)) that allows shock exogeneity to be conditional on some other observed data $\boldsymbol{q}_{m}$ (as well as product characteristics):
Both condition ((ref)) and Assumption (ref) hold when the shocks are unconditionally as-if randomly assigned; the modified assumption will then be helpful to construct more powerful instruments that use information in $\boldsymbol{q}_{m}$ (e.g., lagged market shares and prices as in Section (ref)). In other settings, conditioning on potential confounders in $\boldsymbol{q}_{m}$ may help make shock exogeneity more plausible, such as when the $g_{jm}$ are systematically correlated with some observed $q_{jm}$.
We say $Z_{jm}$ is a vector of formula instruments when it can be written as $Z_{jm}=f_{jm}(\boldsymbol{g}_{m},\boldsymbol{x}_{m},\boldsymbol{q}_{m})$ for some non-stochastic and known vector-valued functions $\left(f_{jm}\right)_{m,j\in\mathcal{J}_{m}}$. We further say that $Z_{jm}$ consists of recentered formula instruments (or just recentered IVs) when the formulas satisfy:
Our first result shows this property characterizes the complete set of valid instruments in our setup:
This result follows immediately from Proposition 1 in borusyak_efficient_2021.\footnote{ Like there, the “only if” part of Lemma 1 should be understood as follows: unless $Z_{jm}$ consists of recentered instruments, it is possible to find a conditional distribution of $\xi_{jm}$ such that Assumption (ref) holds but $\expec{Z_{jm}\xi_{jm}}\ne0$. } It shows that when researchers are only willing to assume that cost shocks are exogenous, in the sense of Assumption (ref), the only justified instruments are recentered IVs. More positively, it shows that any candidate formula instrument $ h_{jm}(\boldsymbol{g}_{m},\boldsymbol{x}_{m},\boldsymbol{q}_{m}) $ for some functions $\left(h_{jm}\right)_{m,j\in\mathcal{J}_{m}}$ can be made a valid IV by recentering: i.e., by subtracting off its conditional mean $ \expec{h_{jm}(\boldsymbol{g}_{m},\boldsymbol{x}_{m},\boldsymbol{q}_{m})\mid\boldsymbol{x}_{m},\boldsymbol{q}_{m}} $.
BH1 and borusyak_design-based_2023 discuss several strategies for recentering formula instruments. One general approach follows when it is possible to generate a set of counterfactual shock vectors $\boldsymbol{g}_{m}^{(c)}$, for $c=1,\dots,C$, which are either drawn from the same distribution as $\boldsymbol{g}_{m}$ (conditional on $\boldsymbol{x}_{m}$ and $\boldsymbol{q}_{m}$) or otherwise as likely to have been realized. For example, the $\boldsymbol{g}_{m}^{(c)}$ could be generated by redrawing from the same randomization protocol that generated $\boldsymbol{g}_{m}$ in a randomized trial, as in the Section (ref) motivating example. In observational data, the counterfactual shocks can instead be generated by reshuffling the set of observed $g_{jm}$ across comparable products, markets, or both. Given such $\boldsymbol{g}_{m}^{(c)}$, any formula instrument $h_{jm}(\boldsymbol{g}_{m},\boldsymbol{x}_{m},\boldsymbol{q}_{m})$ can be recentered by recomputing instrument values under each counterfactual (holding fixed $\boldsymbol{x}_{m}$ and $\boldsymbol{q}_{m}$) and subtracting the average across these values, $ \frac{1}{C}\sum_{c}h_{jm}(\boldsymbol{g}_{m}^{(c)},\boldsymbol{x}_{m},\boldsymbol{q}_{m}) $, for each $j$ and $m$.\footnote{ The number of counterfactuals $C$ does not matter for recentered IV consistency, though it generally affects the asymptotic variance. See footnotes 19 and 21 in BH1. } A second general approach follows when the formula instrument is linear in the shocks: i.e., when $ h_{jm}(\boldsymbol{g}_{m},\boldsymbol{x}_{m},\boldsymbol{q}_{m}) = \sum_{k\in\mathcal{J}_{m}}w_{jkm}g_{km} $ for some exposure weights (or “shares”) $w_{jkm}$ that are functions of $(\boldsymbol{x}_{m},\boldsymbol{q}_{m})$. Recentering such shift-share IVs only requires de-meaning the cost shocks (or “shifts”) by their conditional expectations, $\expec{g_{km}\mid\boldsymbol{x}_{m},\boldsymbol{q_{m}}}$.
Lemma (ref) implies that a recentered IV vector $Z_{jm}$ locally identifies model parameters $\theta$ under a rank condition rothenberg1971identification: that the matrix $\expec{Z_{jm}\nabla_{jm}^{\prime}}$ is full column rank, where
for $ \nabla_{jm}^{\sigma} = \partial\mathcal{D}_{j}\left(\boldsymbol{s}_{m};\sigma,\boldsymbol{x}_{m}^{(1)},\boldsymbol{p}_{m}\right) / \partial\sigma $ with the derivative evaluated at true parameter values.\footnote{ As newey1994large note, establishing global identification with nonlinear moment conditions is generally challenging; the objective function for mixed logit estimation is known to not be globally convex Conlon2020. } Note again that $\theta$ is identified while $\beta$ is not, since recentering makes $Z_{jm}$ uncorrelated with $x_{jm}$. But this is unimportant since $\beta$ does not directly enter price elasticities or important policy counterfactuals. We next discuss how likely powerful IVs can be constructed.
We propose constructing recentered IVs that predict how the vector of model's residual derivatives $\nabla_{jm}$ responds to the exogenous cost shocks. Specifically, we consider an instrument vector of length $\dim(\theta)$ approximating:
The first term of $\tilde{Z}_{jm}$ is the best predictor of the residual derivatives given the cost shocks, product characteristics, and other data in $\boldsymbol{q}_{m}$, where the expectation is taken over the conditional distribution of $(\boldsymbol{p}_{m},\boldsymbol{s}_{m})$ that corresponds to different realizations of unobserved demand and cost shocks. The second term recenters this best predictor by its expectation over the shocks, $ \expec{\expec{\nabla_{jm}\mid\boldsymbol{g}_{m},\boldsymbol{x}_{m},\boldsymbol{q}_{m}}\mid\boldsymbol{x}_{m},\boldsymbol{q}_{m}} = \expec{\nabla_{jm}\mid\boldsymbol{x}_{m},\boldsymbol{q}_{m}} $. Hence $\tilde{Z}_{jm}$ captures how the model residual's derivative is affected by the specific draw of shocks. Note that this $\tilde{Z}_{jm}$ is guaranteed to satisfy the rank condition when cost shocks are relevant (i.e. when $\tilde{Z}_{jm}\neq0$) since then $ \expec{\tilde{Z}_{jm}\nabla_{jm}^{\prime}} = \expec{\tilde{Z}_{jm}\tilde{Z}_{jm}^{\prime}} $ which is generally fully rank.\footnote{ Formally, $ \expec{\tilde{Z}_{jm}\nabla_{jm}^{\prime}} = \expec{\tilde{Z}_{jm}\expec{\nabla_{jm}\mid\boldsymbol{g}_{m},\boldsymbol{x}_{m},\boldsymbol{q}_{m}}^{\prime}} = \expec{\tilde{Z}_{jm}\tilde{Z}_{jm}^{\prime}} $ by the law of iterated expectations and the fact that $ \expec{\tilde{Z}_{jm}\expec{\nabla_{jm}\mid\boldsymbol{x}_{m},\boldsymbol{q}_{m}}^{\prime}} = 0 $ by virtue of recentering. } Below we show $\tilde{Z}_{jm}$ is closely related to the asymptotically efficient recentered IV vector.
The overall logic of our approximations to $\tilde{Z}_{jm}$ is as follows. Instead of integrating over the unobserved shocks, we predict $\nabla_{jm}$ in a single “no-shock” scenario that would prevail in the absence of unexpected $\boldsymbol{g}_{m}$ shocks, i.e. when $ \boldsymbol{g}_{m} = \expec{\boldsymbol{g}_{m}\mid\boldsymbol{x}_{m},\boldsymbol{q}_{m}} $. This scenario is constructed using the data in $(\boldsymbol{x}_{m},\boldsymbol{q}_{m})$ only. We then predict how prices and shares would deviate because of the realized shocks. Specifically: cost shocks affect prices and the researcher can construct an unexpected component of prices using an auxiliary model of cost shock pass-through, as in Section (ref). In turn, prices affect market shares; using preliminary values of demand parameters, the researcher can then measure the impact of unexpected price changes on shares. Finally, $\partial\mathcal{D}_{j}/\partial\sigma$ is a function of the shares so it can be predicted for the scenario with shocks, as a function of $(\boldsymbol{g}_{m},\boldsymbol{x}_{m},\boldsymbol{q}_{m})$. To approximate $\tilde{Z}_{jm}$, it then remains to recenter this prediction as in Section (ref).
The IV construction proceeds in four steps. First, the researcher picks some preliminary values of the parameters $\check{\alpha}<0$ and $\check{\sigma}$. For now, we view these as non-stochastic though it is without loss to allow them to be functions of $\left(\boldsymbol{x}_{m},\boldsymbol{q}_{m}\right)$. We discuss in-sample estimation in Section (ref).
Second, the researcher constructs the no-shock scenario which comprises of a prediction of prices and shares $(\check{\boldsymbol{p}}_{m},\check{\boldsymbol{s}}_{m})$ based on the information in $\left(\boldsymbol{x}_{m},\boldsymbol{q}_{m}\right)$ only. With panel data and persistent cost and demand shocks, a natural choice is the prices and shares in a period prior to the realization of the $\boldsymbol{g}_{m}$ shocks (collected in $\boldsymbol{q}_{m}$). In a single cross-section, predicted prices and mean utilities may be the fitted values from regressing prices and $ \mathcal{D}_{j}\left( \boldsymbol{s}_{m};\check{\sigma},\boldsymbol{x}_{m}^{(1)},\boldsymbol{p}_{m} \right) $ on characteristics, respectively, while predicted shares may follow from the model (i.e., equation ((ref))) at the parameters $\check{\sigma}$ and implied mean utilities.\footnote{ One could also include recentered shocks in these regressions to improve predictive power but take fitted values corresponding to the characteristics only. Nonlinear predictions, such as with machine learning algorithms, may improve precision, too. }
Third, the researcher forms price predictions as deviations from $\check{\boldsymbol{p}}_{m}$ due to the exogenous cost-shocks. For clarity here we work with the simplest predictions:
where $\tilde{g}_{jm}=g_{jm}-\expec{g_{jm}\mid\boldsymbol{x}_{m},\boldsymbol{q}_{m}}$ is product $j$'s recentered cost shock and $\check{\pi}\ne0$ is a pass-through coefficient that is again assumed non-stochastic or measurable with respect to $(\boldsymbol{x}_{m},\boldsymbol{q}_{m})$, for now. These predictions need not be correct for the resulting instruments to be valid, as in Section (ref). We consider extensions with more elaborate shock pass-through models below.
The price prediction immediately suggests the first instrument (recall equation ((ref))): $-\check{\pi}\tilde{g}_{jm}$, or equivalently the recentered shock $\tilde{g}_{jm}$. The price prediction also implies a prediction for mean utilities that will shortly prove helpful: \[ \hat{\delta}_{jm} = \check{\delta}_{jm}+\check{\alpha}\check{\pi}\tilde{g}_{jm} \quad\text{for } \check{\delta}_{jm} = \mathcal{D}_{j}\left( \check{\boldsymbol{s}}_{m};\check{\sigma},\boldsymbol{x}_{m}^{(1)},\check{\boldsymbol{p}}_{m} \right). \]
Finally, the researcher generates predictions for the market shares and ultimately $\nabla_{jm}^{\sigma}$. We propose two versions: a first-order approximation which yields a recentered shift-share IV, and an exact prediction that may yield a more powerful instrument but generally requires further recentering. Using the first-order approximation, one predicts market shares as
and the model residual's derivative with respect to the vector $\sigma$ as
Here $ \check{\nabla}_{jm}^{\sigma} = \frac{\partial}{\partial\sigma} \mathcal{D}_{j}\left(\check{\boldsymbol{s}}_{m};\check{\sigma},\boldsymbol{x}_{m}^{(1)},\check{\boldsymbol{p}}_{m}\right) $ predicts the residual's derivative in the absence of shocks and
is the predicted first-order effect of $\tilde{g}_{km}$ on that derivative. Recentering the prediction in ((ref)) eliminates the first term, resulting in the $\dim(\theta)\times1$ vector of shift-share instruments \[ Z_{jm}^{SSIV} = \left(
\right). \] Again, an advantage of $Z_{jm}^{SSIV}$ is that it only requires specification of the conditional shock means $\expec{\boldsymbol{g}_{m}\mid\boldsymbol{x}_{m},\boldsymbol{q}_{m}}$ for recentering. Also conveniently, the choice of $\check{\pi}$ is immaterial with this approach, as it only rescales the instruments. Without a random coefficient in price the first term in equation ((ref)) drops out, making $\check{\alpha}$ immaterial too.
Alternatively, the researcher can obtain the exact prediction of how changes in mean utilities due to the cost shocks affect shares: \[ \hat{s}_{jm} = \mathcal{S}_{j}(\hat{\boldsymbol{\delta}}_{m};\check{\sigma},\boldsymbol{x}_{m}^{(1)},\hat{\boldsymbol{p}}_{m}), \] and how those changes affect $\nabla_{jm}^{\sigma}$: \[ \hat{\nabla}_{jm}^{\sigma} = \frac{\partial}{\partial\sigma} \mathcal{D}_{j}\left(\hat{\boldsymbol{s}}_{m};\check{\sigma},\boldsymbol{x}_{m}^{(1)},\hat{\boldsymbol{p}}_{m}\right). \] This $\hat{\nabla}_{jm}^{\sigma}$ is a nonlinear function of the shocks which needs to be recentered, generally by specifying shock counterfactuals as described in Section (ref).\footnote{ This $\hat{\nabla}_{jm}^{\sigma}$ relates to the efficient IV construction of Berry1999: the two would be equivalent if the price predictions $\hat{p}_{jm}$ were taken from an equilibrium pricing model and mean utilities were set to $ \hat{\delta}_{jm} = \check{\alpha}\hat{p}_{jm}+\check{\beta}^{\prime}x_{jm} $ for an initial value $\check{\beta}$ of $\beta$. Our instrument differs in three ways: it is based on a shock pass-through model that need not be correct, it uses additional information (in particular, when lagged prices and shares are available), and it is recentered to avoid bias from endogenous characteristics. Conlon2020 propose an improvement on the Berry1999 instrument that integrates over an empirical distribution of $\xi_{jm}$ rather than setting unobserved demand shocks to zero. Our approach could be similarly extended. } The recentered formula IV vector is then: \[ Z_{jm}^{FIV} = \left(
\right). \]
Appendix Proposition (ref) builds intuition for these instruments by considering the case where the nonlinear parameters $\sigma$ are the standard deviations of random coefficients and $\check{\sigma}$ is close to zero, which would correspond to a pure multinomial logit model.\footnote{ salanie_fast_2022 use an approximation of mixed logit around $\sigma=0$ to simplify share inversion and estimation. Our proof of Appendix Proposition (ref) offers a new derivation that yields additional intuition, discussed in Appendix (ref). } In this “local-to-logit” approximation, the shift-share IV corresponding to the standard deviation $\sigma_{\ell}$ of a non-price characteristic $x_{jm\ell}$ can be written, up to a scaling factor, as: \[ z_{jm\ell}^{SSIV} \approx x_{jm\ell}\cdot\sum_{k\in\mathcal{J}_{m}}\check{s}_{km}\left(x_{km\ell}-\bar{x}_{m\ell}\right)\tilde{g}_{km} \quad\text{for } \bar{x}_{m\ell} = \sum_{k\in\mathcal{J}_{m}}\check{s}_{km}x_{km\ell}. \] This IV an interaction between product $j$'s own characteristic $x_{jm\ell}$ and a market-specific aggregate of the shocks: the share-weighted covariance across the products in the market between $x_{km\ell}$ and the recentered cost shock (including the outside good with a shock set to zero). The covariance is positive when the cost shocks unexpectedly make products with high $x_{km\ell}$ more expensive relative to other products in the market. Thus, $z_{jm\ell}^{SSIV}$ is performing an analysis similar to difference-in-differences: it compares changes in market shares for high-$x_{jm\ell}$ vs. low-$x_{jm\ell}$ products in markets where high-$x_{km\ell}$ vs. low-$x_{km\ell}$ products became less competitive because of the exogenous cost shocks. Identification with this instrument is therefore based on the core property of mixed logit models: that, after a cost shock, market shares are reallocated towards products with similar characteristics when $\sigma_{\ell}$ is large, but to all products evenly (in proportion of their market shares) when $\sigma=0$. The instrument for the random coefficient in price has additional terms related to how cost shocks affect prices directly; see Appendix (ref).
We use a generalized method of moments (GMM) procedure to estimate $\theta$. Specifically, we generalize the moment condition ((ref)) to write:
where $Z_{jm}\left(\check{\theta},\check{\pi}\right)$ is a vector of recentered instruments of the same dimensionality as $\theta$, constructed as above, now with the dependence on preliminary parameter values $(\check{\theta},\check{\pi})$ made explicit. The $\mathcal{B}_{j}(\boldsymbol{x}_{m},\boldsymbol{q}_{m};\gamma,\check{\theta})$ term is included to reduce residual variation in the error $\Xi_{jm}=\beta'x_{jm}+\xi_{jm}$. Since recentered IVs are uncorrelated with any function of $(\boldsymbol{x}_{m},\boldsymbol{q}_{m})$, the moment conditions ((ref)) hold at the true parameter values regardless of $\mathcal{B}_{j}(\cdot)$ and for any $\gamma$.
Two examples of $\mathcal{B}_{j}(\cdot)$ are illuminating. First, it can be set to $\gamma'x_{jm}$ with $\gamma$ estimated as a projection coefficient (i.e., from regressing the estimate of $\Xi_{jm}$ on $x_{jm}$). While $\gamma$ need not coincide with the causal effect of characteristics on demand, $\beta$, this is not a problem for many important policy counterfactuals (as discussed in Section (ref)). Second, in panel data, one can set $\mathcal{B}_{j}$ to be the lagged value of $\Xi_{jm}$ obtained from the lagged prices and shares using the initial parameter values, $\check{\theta}$ (and with no additional parameters, $\gamma=\emptyset$). This corresponds to estimating the model “in differences,” as commonly done in linear models and sometimes also for nonlinear demand models (e.g., Adao2017). Such differencing helps reduce residual variation when the unobserved demand shifters are strongly serially correlated.
Our baseline estimator is the $(\hat{\theta},\hat{\gamma})$ that solves the sample analog of condition ((ref)), averaging across product-market pairs, along with the sample analog of a moment condition for $\gamma$:
This condition defines $\gamma$ as the coefficient giving the least-squares fit of $\Xi_{jm}$ on $\mathcal{B}_{j}(\boldsymbol{x}_{m},\boldsymbol{q}_{m};\gamma,\check{\theta})$ (e.g. a projection on $x_{jm}$ in the above linear case). It can be understood in the same way as how, in regression analyses of randomized control trials, predetermined controls are included to soak up residual outcome variation with the coefficients on them not interpreted causally. Here we assume the researcher has obtained initial values of $\theta$ by, say, an initial GMM procedure with $\tilde{g}_{jm}$ and conventional characteristic-based IVs as instruments. The researcher has also obtained an initial pass-through constant $\check{\pi}$, for example from least-squares estimation of $p_{jm}=\pi_{0}+\pi\tilde{g}_{jm}+\epsilon_{jm}$.
We also consider a “continuously updating” estimator which, unlike the baseline estimator, does not require initial values of $\theta$. This estimator replaces $\check{\theta}$ in the moment conditions ((ref))–((ref)) with $\theta$; that is, the instruments $Z_{jm}$ (and, if applicable, the $\mathcal{B}_{j}$ term) are updated when searching for the parameter estimate.\footnote{ We use the term “continuously updating” differently to the standard continuously updating estimator, where it refers to the choice of the GMM weighting matrix. That choice is not relevant to our just-identified setting. } We use this estimator in our simulations, below. Another alternative is to use a two-step or iterative GMM procedure.
Consistency and asymptotic normality of these estimators follow from standard GMM theory (e.g. newey1994large) when there are many iid markets $m$ (or, more generally, many iid market clusters: e.g., in a panel with many regions and a small number of time periods). In other cases, such as when there are only a few markets or when across-market linkages create dependences in the instruments and GMM residuals, asymptotic properties can be established from many iid shocks $g_{jm}$ (or, more generally, many iid shock clusters) following adao2019shift, BJH2018, and BH1. This strategy is helpful, for instance, when the markets represent regions and the shocks arise from exchange rate fluctuations, which affect all regions at once. Other shocks can affect the demand for similar products across multiple regions, too. We develop this approach in a setting where $\gamma=\emptyset$ (e.g. the differencing case discussed above), $Z_{jm}$ consists of shift-share instruments, and the preliminary values $(\check{\theta},\check{\pi})$ are non-stochastic.\footnote{ For recentered instruments that do not have a shift-share structure, BH1 provide sufficient conditions for consistency in linear IV settings. Adapting them to our current setting is left to future work. No general asymptotic inference results are currently known for such instruments, even for linear IV settings. } The estimator $\hat{\theta}$ then solves:
where $ \mathcal{R}_{km}(\hat{\theta}) = \sum_{j\in\mathcal{J}_{m}}w_{jkm} \left( \mathcal{D}_{j}\left(\boldsymbol{s}_{m};\hat{\sigma},\boldsymbol{x}_{m}^{(1)},\boldsymbol{p}_{m}\right) - \hat{\alpha}p_{jm} - \mathcal{B}_{j}(\boldsymbol{x}_{m},\boldsymbol{q}_{m},\check{\theta}) \right) $ is an “aggregated” shock-level residual in the sense of adao2019shift and BJH2018.\footnote{ Note that we include the “price instrument” $-\check{\pi}\tilde{g}_{jm}$ as a shift-share IV here, with $w_{jkm}=-\check{\pi}\mathbf{1}[j=k]$. } Equation ((ref)) represents $\hat{\theta}$ as the solution of a “shock-level” GMM procedure, with an estimable variance of $\sum_{m}\sum_{k\in\mathcal{J}_{m}}\tilde{g}_{km}\mathcal{R}_{km}(\theta)$ given many iid shocks or many shock clusters that allow for shock correlations across markets. Standard GMM expressions can then be applied, as before, regardless of the correlation structure in the residual $ \mathcal{D}_{j}\left(\boldsymbol{s}_{m};\sigma,\boldsymbol{x}_{m}^{(1)},\boldsymbol{p}_{m}\right) - \alpha p_{jm} - \mathcal{B}_{j}(\boldsymbol{x}_{m},\boldsymbol{q}_{m},\check{\theta}) $ across both products and markets. Convergence of $\hat{\theta}$ only requires the shocks to induce sufficient variation across product-market pairs in the IVs.
Results on the asymptotic efficiency of recentered IVs for linear structural equations, developed by borusyak_efficient_2021, can also be extended to characterize the optimal IV matrix, $Z^{*}=\left(Z_{jm}^{*\prime}\right)_{m,j\in\mathcal{J}_{m}}$ without assuming many independent markets. Under appropriate regularity conditions, it takes the form:
where $\xi=\left(\xi_{jm}\right)_{m,j\in\mathcal{J}_{m}}$, $\nabla=\left(\nabla_{jm}^{\prime}\right)_{m,j\in\mathcal{J}_{m}}$, and $\boldsymbol{v}=\left(\boldsymbol{v}_{m}\right)_{m}$ for any variable $\boldsymbol{v}$. The inner term in parentheses stacks the recentered best predictors of the model's residual derivatives, $\tilde{Z}_{jm}$. The recentered predictor is then adjusted by $\expec{\xi\xi^{\prime}\mid\boldsymbol{x},\boldsymbol{q}}^{-1}$, which can be understood as combining a partial residualization of $\tilde{Z}_{jm}$ on $\expec{\xi\mid\boldsymbol{x},\boldsymbol{q}}$ and a reweighting by $\var{\xi\mid\boldsymbol{x},\boldsymbol{q}}^{-1}$ (see Proposition 3 in borusyak_efficient_2021). This characterization provides further motivation for our focus on approximating $\tilde{Z}_{jm}$ as well as for the adjustment for $\mathcal{B}_{j}(\boldsymbol{x}_{m},\boldsymbol{q}_{m},\check{\theta})$ in ((ref)), as a proxy for $\expec{\xi_{jm}\mid\boldsymbol{x},\boldsymbol{q}}$, in estimation. Weighting by an estimate of the residual's inverse variance, as in feasible generalized least squares, is less popular in practice and not pursued here.
We now develop several extensions of the baseline model. We consider them one by one to avoid notational clutter, but in practice they can be combined.
\paragraph{Observed Consumer Characteristics.}
In some applications, the researcher observes the distribution of consumer characteristics in each market and allows these consumer characteristics correlate with tastes for product characteristics in $\boldsymbol{x}_{m}^{(1)}$. Our results extend immediately to that case. Specifically, the consumer with characteristics $c_{i}=\left(c_{ir}\right)_{r=1}^{R}$ solves: \[ \max_{j\in\mathcal{J}_{m}\cup\left\{ 0\right\} } \delta_{jm} + \left(\sum_{r=1}^{R}\gamma_{r0}c_{ir} + \eta_{i0}\right)p_{jm} + \sum_{\ell=1}^{L_{1}} \left(\sum_{r=1}^{R}\gamma_{r\ell}c_{ir}+\eta_{i\ell}\right) x_{jm\ell}+\varepsilon_{ijm}, \] where $\gamma=\left(\gamma_{r\ell}\right)$ serve as additional nonlinear parameters and extreme-value shocks $\varepsilon_{ijm}$ are independent from $(c_{i},\eta_{i})$. The distribution $\mathcal{\mathcal{P}}_{m}\left(\cdot;\sigma\right)$ of $(c_{i},\eta_{i})$ is known: typically $c_{i}$ is assumed independent of $\eta_{i}$ with the market-specific distribution taken from the data. Again, the model is invertible (see, e.g., Gandhi2021), and the rest of the analysis goes through without change.
\paragraph{Using Lagged Prices and Shares with Product Entry and Exit.}
When lagged prices and shares are available, our baseline recommendation is to use them as $\left(\check{\boldsymbol{p}}_{m},\check{\boldsymbol{s}}_{m}\right)$ when constructing the instruments. This approach requires a modification when some products have recently entered the market and their lagged information is not available. Moreover, if many products have exited, lagged shares may be a poor prediction of the current period's share in the absence of cost shocks.
In such cases, our proposal is to predict prices $\check{p}_{jm}$ and mean utilities $\check{\delta}_{jm}$ for all products in the current period and construct shares from them, as $ \check{s}_{jm} = \mathcal{S}_{j}(\check{\boldsymbol{\delta}}_{m};\check{\sigma},\boldsymbol{x}_{m}^{(1)},\check{\boldsymbol{p}}_{m}) $. For continuing products, lagged price can serve as $\check{p}_{jm}$, while mean utility can be obtained from the inversion of lagged shares, given $\check{\sigma}$. For new products, one may proceed as in a single cross-section, taking fitted values from regressions of realized price and implied mean utility $ \mathcal{D}_{j}\left(\boldsymbol{s}_{m};\check{\sigma},\boldsymbol{x}_{m}^{(1)},\boldsymbol{p}_{m}\right) $ on characteristics.
\paragraph{Identification of $\beta$ via Instruments for Product Entry.}
Our baseline analysis assumes that the $\boldsymbol{g}_{m}$ shocks do not affect the characteristics of available products, which leaves the causal effect of characteristics on mean utility, $\beta$, unidentified. This is in contrast to the price coefficient $\alpha$ which is identified because price is affected by the shocks and the researcher is able to construct a relevant recentered instrument, $\check{\pi}\tilde{g}_{jm}$. If the researcher has access to shocks that affect some characteristics in a predictable way, those characteristics can be treated in the same way as the baseline model treats price, and the corresponding components of $\beta$ become identified.
\paragraph{Incorporating a Pricing Model.}
Our baseline analysis focuses solely on demand estimation, leaving the supply side flexible. Although we require an auxiliary model of cost shock pass-through, it need not be correct and is indeed very simple in our baseline proposal. This modular approach has the advantage that demand can be estimated with fewer assumptions. However, some counterfactuals, such as a merger between two firms, require predicting how prices would change, and thus taking a stand on how firms set prices. In that case, the researcher may consider leveraging the pricing model to obtain more powerful—albeit less robust—demand estimates, too, as in Berry1995. While we leave the technical presentation to future drafts, this extension should be straightforward.
\paragraph{Estimated Mapping from Inputs to Products.}
Our baseline analysis assumes that the cost shocks are product-specific. When the shocks originate from input prices or exchange rate fluctuations, this requires (at least partial) knowledge of the mapping from inputs to products: e.g., the percentages of wheat and corn among the ingredients of ready-to-eat cereals barahona2023equilibrium or the country of assembly for each car model Grieco2021. However, in some settings where input prices are observed, the mapping to products is not available. Villas-Boas2007 addresses this problem by using interactions between market-specific input prices and product dummies as instruments. The intuition is that, when the same product is observed in sufficiently many markets (e.g., time periods), the sensitivity of each product's price to all inputs prices is revealed.
This insight can be adapted to our setting, in a two-step approach. First, the exposure of each product to the set of inputs is estimated by a product-specific (e.g., time-series) regression of price on recentered input price shocks. This regression should have sufficiently many observations per product, such that the parameters converge to some pseudo-true values, which need not reflect the true production function. Second, a product-specific cost shock is constructed as a shift-share aggregate of input price shocks as shifts with estimated exposures as shares and used in the rest of the analysis. We leave the precise asymptotic analysis of this approach to future drafts.
\paragraph{Alternative Demand Models.}
While we have focused on nested and mixed logit, our approach extends to other popular parametric demand systems. Most directly, nested and mixed constant elasticity of substitution (CES) models are closely related, with prices replaced with log prices and quantity shares replaced with expenditure shares. Our instrument construction then goes through. Similarly, our analysis extends directly to variations on the mixed logit model used in IO, such as the Hotelling model of spatial product differentiation houde_spatial_2012 and the “principles of differentiation” model of bresnahan1997market which combines multiple nest groupings.
\paragraph{Non-Parametric Demand.}
We follow Erry2014 in considering non-parametric identification of demand under Assumption (ref) instead of the conventional stronger assumption (our equation ((ref))) that they impose. As they show, identification requires an index restriction: that at least one component of $(p_{jm},x_{jm}')'$ enters demand without a random coefficient. Moreover, exogenous cost shocks are only sufficient for non-parametric identification if price satisfies this property. We show that this result extends to our weaker assumption, too:
We now analyze the bias and variance properties of the recentered IV approach, relative to conventional alternatives, in a Monte Carlo simulation that largely follows Gandhi2015. Section (ref) describes the baseline data-generating process, where both conventional and recentered IVs are valid, and shows what data features drive the variance of estimates in the two approaches. Section (ref) then introduces product characteristic endogeneity, demonstrating that our proposed IVs remain accurate while conventional characteristic-based IVs are significantly biased. Details of the algorithms used for estimation are reported in Appendix (ref).
\paragraph{Simulation Design.}
We simulate a set of regions $r=1,\dots,100$ in two periods $t\in\left\{ 1,2\right\} $; hence $m=(r,t).$ In each period, consumers choose between products $j\in\mathcal{J}_{m}=\{1,\ldots,15\}$ and the outside good to maximize their utility, according to equation ((ref)). We consider $L_{1}=2$ observed time-invariant characteristics $x_{jr\ell}\stackrel{iid}{\sim}N(0,1)$, in addition to the intercept $x_{jr0}=1$. Random coefficients are placed on both characteristics, $x_{jm}^{(1)}=\left(x_{jm1},x_{jm2}\right)$ but not on price. The random coefficients $\eta_{i\ell}\stackrel{iid}{\sim}N(0,\sigma_{\ell}^{2})$ have true standard deviations of $\sigma_{\ell}=4$ for $\ell=1,2$. Product $j$'s mean utility $\delta_{jm}$ is determined each period according to equation ((ref)) with persistent unobserved demand shifters: $\xi_{jr1}\stackrel{iid}{\sim}N(0,1)$ and $\xi_{jr2}=0.9\xi_{jm1}+\sqrt{1-0.9^{2}}\cdot e_{jm}$ for $e_{jm}\stackrel{iid}{\sim}N(0,1)$. We set $\beta_{0}=35$, $\beta_{1}=\beta_{2}=2$, and $\alpha=-0.2-4\exp(0.5)$. Market shares are simulated with 1,000 independent draws:
Prices are set by a simultaneous Bertrand-Nash game where each product is produced by a single firm. In each period the price vector for each region $\boldsymbol{p}_{mt}$ is the solution to the following system of equations derived from the firms' first-order conditions:
where $c_{jm}=\gamma^{\prime}x_{jm}+\omega_{jm}+g_{jm}$ is firm $j$'s marginal cost (and the derivative with respect to price includes the effect through $\boldsymbol{\delta}_{m}$). We set $\gamma_{0}=5$ and $\gamma_{1}=\gamma_{2}=1$ and generate persistent unobserved cost shocks: $\omega_{jr1}\stackrel{iid}{\sim}N(0,1)$ and $\omega_{jr2}=0.9\omega_{jr1}+\sqrt{1-0.9^{2}}\cdot w_{jr}$ for $w_{jr}\stackrel{iid}{\sim}N(0,1)$. The observed cost shocks only happen in the second period, such that $g_{jr1}=0$ and $g_{jr2}\stackrel{iid}{\sim}N(0,0.2^{2})$.\footnote{ The parameters of our simulation were picked to follow the simulation in Gandhi2015 as much as possible. The deviations arise for three reasons: we have two periods, we distinguish between observed and unobserved costs shocks, and we do not have a random coefficient on price. Our value for $\alpha$ is picked as the average price coefficient: they set the linear coefficient on price to $-0.2$ and have (the negative of) log-normal random coefficients with the mean $-4e^{0.5}$. We also set $L_{1}=2$ instead of $4$ and $\beta_{0}=35$ instead of $50$. } Note that the variance of $g_{jr2}$ is only 4% of the variance of unobserved cost shocks, consistent with the limited exogenous shock variation expected in typical applications.
We estimate this model for two alternative sets of moment conditions. For the conventional characteristic-based IVs, let $Z_{jr2}^{C}$ be a vector collecting $g_{jr2}$, $x_{jr}$, and a set of two instruments for $\sigma=(\sigma_{1},\sigma_{2})$: either BLP (sum of competitor characteristics) instruments or the local or quadratic differentiation IVs for proposed by Gandhi2015. These instruments are given by:
with a proximity threshold $\kappa_{\ell}$; we follow Gandhi2015 and use the standard deviation of $x_{jm\ell}$. We then estimate $(\alpha,\beta,\sigma)$ via GMM using data from the second period only (when the cost shock is available) and the moment condition:
For the recentered instruments, let $Z_{jr2}^{R}$ be a vector collecting $g_{jr2}$ and either the shift-share or recentered exact prediction IVs proposed in Section (ref). We recenter by permuting the cost-shocks 20 times across both products and markets. We use the continuously updating procedure proposed in Section (ref) to bypass the need for initial estimates (iterative GMM yields very similar results). The moment condition is:
for $\Delta\xi_{jr}=\xi_{jr2}-\xi_{jr1}$. Differencing corresponds to setting $\mathcal{B}_{j}$ defined in Section (ref) to the lagged values $\beta'x_{jr}+\xi_{jr1}$ (since characteristics are time-invariant they drop from the estimation) which helps reduce residual variation since the demand shifters are serially correlated. Since we simulate a panel with two periods, it is natural to set the prediction for prices and shares that we will use to construct $Z_{jr2}^{R}$ to their pre-cost shock (i.e. period one) values, $ (\check{p}_{jr2},\check{s}_{jr2}) = (p_{jr1},s_{jr1}) $. See Appendix (ref) for additional details on our estimation.
\paragraph{Baseline Results.}
Figure (ref) shows the results of our estimation for 100 Monte Carlo simulations. As expected, each set of instruments yields approximately unbiased estimates for each of the parameters. The recentered IVs tend to estimate the price coefficient somewhat more precisely than the differentiation IVs, with a tighter distribution of estimates, while the reverse is true for the nonlinear parameters. BLP instrument estimates are considerably noisier for all parameters; we drop them going forward to focus on the leading characteristic-based IVs.
\paragraph{Sensitivity.}
We next study how the precision of the recentered IV and differentiation IV estimates varies with two key features of the data-generating process. Figure (ref) shows that recentered IVs have less power to estimate the nonlinear parameters $\sigma$ with a lower variance of cost shocks (both IV approaches benefit from more variable shocks for estimating the price coefficient). In turn, Figure (ref) shows that differentiation IVs have lower power for $\sigma$ with less variation in choice sets across markets. While in our baseline simulation each market has an independent draw of product characteristics, here we make a subset of products the same across all markets, as when sold nationally. In the extreme case where all products are common across markets, differentiation IVs only have variation because product fixed effects are not included in our estimation procedure.
We now show that our proposed IVs continue to accurately estimate the price elasticity parameters even when product characteristics are endogenous, while the differentiation IVs do not. We consider a simple model of characteristic endogeneity that assumes each region has a time-invariant “bliss point” $B_{r}$ for the first product characteristic.\footnote{This simulation is in spirit to Gandhi2015, Section 4.4.} Consumers dislike products far from the bliss point, which we model by subtracting $3(x_{jr1}-B_{r})^{2}$ from $\xi_{jrt}$. Realizing this, firms introduce more products near the bliss point, which we model by centering the distribution of $x_{jr1}$ around $B_{r}$: $x_{jm1}\stackrel{iid}{\sim}N(B_{m},1)$. Here the differentiation IVs of Gandhi2015 are invalid because popular products are in the dense part of the distribution of product characteristics. By contrast, our proposed IVs only require exogeneity of the cost shocks.
Figure (ref) summarizes the results: there is little bias when using our proposed IV and a moderate decrease in power relative to Figure (ref). However, using either of the two differentiation IV strategies yields substantially biased estimates for $\sigma_{1}$: the estimates are equal to zero in all simulation draws. The reason is that, under exogenous entry, mixed logit predicts a negative correlation between market shares and the degree of local competition as proxied by the differentiated IVs. This negative correlation is especially strong when the variance of random coefficients is high. However, with endogenous entry, as in our simulations, market shares tend to correlate positively with being in the dense part of the characteristic space (because that means being close to the bliss point). This generates a strong negative bias in $\sigma_{1}$ when using the differentiation IVs.\footnote{ We thank Jean-Francois Houde for pointing this out to us. } Notably, this bias seems to also affect differentiation IV estimation of the price coefficient $\alpha$ and the other nonlinear parameter $\sigma_{2}$; recentered IV estimates remain unbiased for these parameters as well.
Modern demand models give a flexible yet tractable structure for substitution across goods. We develop new tools for bringing this structure to data by leveraging its predictions of how key endogenous variables respond to a set of exogenous supply-side shocks. Our recentered IV approach avoids the widespread but often implausible assumption of exogenous product characteristics, letting us “reuse” the exogenous shocks to construct multiple powerful instruments targeted at each of the nonlinear parameters of the model. Simulations suggest recentered IVs can have comparable power to leading characteristic-based IVs while avoiding severe bias from characteristic endogeneity. Future drafts will illustrate this approach in a real setting.
Several open paths remain in this agenda. First, we have only considered here demand estimation with market-level data; the role of recentered IV with individual choice-level data is an interesting question for future research. Second, while we have characterized non-parametric identification of demand with recentered IVs, flexible estimation (using, e.g., modern machine learning tools) is worth further study. Finally, we expect recentered IVs to be useful for identifying structural models beyond demand—such as for dynamic choice or strategic entry in IO, or other phenomena in macroeconomics, international trade, and spatial economics. Developing these extensions may yield practical new ways to improve the credibility and transparency of structural estimation.