Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
33,647 characters · 6 sections · 12 citation commands
Panel Data Estimation of Individual Demand in Markets with Many Consumers
\onehalfspacing
It has long been known that fixed effects estimation controls for time-invariant endogeneity in panel data for individuals. The purpose of this paper is to consider whether and how panel data can be used to estimate individual demand effects while controlling for endogeneity resulting from prices being determined in markets. We find that fixed effects estimation of price effects for individual demand is approximately correct when the number of consumers in a market is large and idiosyncratic (individual-specific, time-varying) heterogeneity is independent of unobserved supply disturbances. Such independence seems plausible in many settings where consumers in a market are distinct geographically or professionally from those who supply that market.
We obtain these results for a demand model similar to kennan1989simultaneous where market demand is the sum of demands of individuals facing a common market price that is determined in equilibrium where market demand equals market supply. We allow there to be many more consumers in a market than are present in the data. Approximate consistency holds because each individual's contribution to the equilibrium price becomes negligible as the number of individuals in the market grows large. This means prices are approximately exogenous from any single consumer's perspective, even though they are endogenous at the market level. We derive the rate at which bias in individual demand estimates shrinks as the number of individuals in the market increases.
We innovate in considering panel data for individual consumers where market prices vary over time. The key insight is that time variation in the market prices, combined with idiosyncratic preferences that are independent of supply shocks, leads to fixed effects estimation being approximately correct.
Independence between idiosyncratic preferences and supply shocks could be a concern if macroeconomic time effects were common to both. Time effects may be controlled for by including observable time variables such as trends or time period dummies. There are some limits to controlling for time effects, for example, in losing identification when there are no restrictions on how demand varies across time and individuals \citep*{hoderlein2012nonparametric, chernozhukov2013average}. As in many other settings, including time period dummies could provide a plausible intermediate approach that allows identification of important properties of individual demand from panel data. We find that with two-way fixed effects (individual and time), estimation is approximately correct when individuals are sampled from different markets.
We first consider a linear model of individual demand with additive individual fixed effects and idiosyncratic disturbances. In this well-understood setting, it is easy to see how independence of supply and idiosyncratic demand shocks provides identifying information while fixed individual effects allow for endogeneity of prices. We then generalize these results to allow for fixed time period effects in individual demand and to allow for a random coefficient on price. In all settings we consider a linear average supply function for tractability. The results of this paper can be extended to general supply functions, a direction we are currently pursuing, but the extension requires slightly different proof techniques. The results can also be easily extended to allow for additional additive covariates in the demand and supply models, but we omit them in this paper for notational simplicity.
Throughout the paper we characterize the rate at which bias of the fixed effects estimator vanishes as the number of individuals in a market grows, rather than seeking strict identification. When market demand is the sum of individual demands, each individual's influence on the equilibrium price always results in a small but nonzero bias. We show that this bias shrinks as the number of consumers in a market increases and characterize the rate at which it shrinks, so that individual price effects are approximately identified from panel data.
This paper motivates the approach taken to estimating price effects using scanner data in chernozhukov2019demand. That paper relied on individual-specific coefficient models to estimate average equivalent variation and deadweight loss from a panel of consumers. The results given in our paper suggest that this approach may be approximately correct when idiosyncratic preferences are independent of supply shocks. This paper also motivates the approach of dubois2020well and lanier2023estimating. In their model the idiosyncratic preferences are i.i.d. Logit (Type I extreme value) disturbances. If the assumption that the idiosyncratic preferences are independent of supply shocks is added, the results given in our paper suggest approximate correctness of their approach, although for a different model of demand. We emphasize that these are conjectures at this time but initial results are promising.
This paper relates to several strands of literature. A large literature in industrial organization identifies and estimates market-level demand using instruments \citep*{berry1995automobile, petrin2002quantifying, berry2004differentiated, berry2014identification, berry2024nonparametric}. We innovate in showing that for panel data on individuals price effects can be identified when idiosyncratic preferences are independent of unobserved supply shocks, without instruments. This result also differs from that of mackay2025estimating, who show that with covariance restrictions, market-level demand can be identified and estimated. Here only idiosyncratic preferences are required to be independent of unobserved supply effects, while time-constant unobserved preferences may still be correlated with supply shocks. This is a different restriction from requiring market demand and supply unobservables to be uncorrelated.
A similar comparison can be made with an older literature on identifying structural coefficients under certain covariance restrictions on the structural errors in the models \citep*{hausman1983identification, hausman1987efficient}. Panel data allows us to modify the zero covariance restriction to only apply to idiosyncratic preferences rather than to the whole structural error.
In the econometrics literature, hoderlein2014nonparametric use panel data to estimate market-level demand, where each individual unit is a market. hausman2016individual and chernozhukov2025fisher estimate functionals of individual demand using panel data, where the individual unit is a consumer. While these methods do not consider price endogeneity, we motivate these methods by showing price endogeneity can be small under certain conditions.
Section (ref) of the paper gives results for a linear panel demand model without a time effect and Section (ref) for a linear panel demand model with a time effect. In Section (ref) we derive results for a random coefficient demand model without a time effect and in Section (ref) we derive results for a random coefficient demand model with a time effect. Proofs of results are deferred to the Appendix.
In this section, we consider a linear demand model and illustrate why, in panel data, when estimating individual demand, bias caused by price endogeneity might disappear with the number of consumers in the market if time differences of individual demand and aggregate supply shocks do not covary and preference variation is uncorrelated across individuals. We suppose price is determined by linear market supply and individual demand equations. We give conditions for time differences in the market price to be nearly uncorrelated with idiosyncratic preference differences, so that a regression of quantity differences on market price differences would be approximately consistent.
Consider a single market with $M$ consumers $i = 1, \dots M$. For a sample of size $N$ of those consumers, we observe quantity $q_{it}$ and price $p_t$ of a single good for each consumer $i$ over $T \geq 2$ time periods $t = 1, \dots, T$. Suppose we can write demand for every consumer in the market as a linear function of price,
where $\alpha_i$ is unobserved time-invariant individual heterogeneity and $\eta_{it}$ is unobserved idiosyncratic individual heterogeneity. Then average demand in each time period $t$, where the average is taken over all individuals in the market, is given by
Suppose, as in kennan1989simultaneous, that we can write the inverse supply model as a linear model of average supply,
for $\varepsilon_t$ an unobserved supply shock. Solving for equilibrium price $p_t$ by setting demand equal to supply then gives
As usual for panel data we can difference these equations to eliminate the individual effect, obtaining
where $\Delta$ denotes the time difference. This is a model of linear supply and demand that possesses the classic price endogeneity problem. Due to the simultaneity of the system, price $p_t$ is correlated with individual unobserved preferences $\alpha_i$ and $\eta_{it}$. This dependence could cause naive estimates of the individual demand equation that do not exploit instrument-like conditions to be inconsistent. However, in what follows, we show that this inconsistency will disappear as the number of consumers in the market $M$ grows, as long as time differences in supply shocks are uncorrelated with time differences in individual idiosyncratic preference shocks $\eta_{it}$. In particular, the inconsistency will disappear at the rate $O(M^{-1})$.
The key condition for the inconsistency to vanish as $M$ grows is the following one.
Assumption (ref) states that the time difference of the supply shock $\Delta\varepsilon_t$ is orthogonal to the time difference of the preference shock and that the preference shocks for different individuals are uncorrelated with each other. We will additionally assume for notational simplicity that $Var(\Delta \eta_{it})$ is common across $i$ and $t$ and $Var(\Delta p_t)$ is common across $t$. Under this condition,
Then we immediately obtain the following result:
In this result we assume that there is sufficient independent variation in $\Delta p_t$ across observations to make the usual formula for the probability limit of OLS valid. Such variation could come from $T$ growing or from $T$ fixed and sampling consumers from independent markets with common $Var(\Delta p_t)$ and $M$. From this formula we see that as the number of consumers $M$ grows $plim(\hat\beta)$ converges to the true $\beta$. In this result we think of taking $N,T \to \infty$ but keeping $M$ fixed as an asymptotic approximation of a large sample size. While we can also perform a finite-sample analysis of $\hat\beta$ and bound the size of the bias, we will not pursue this for the linear demand and supply models studied in this paper.
This result demonstrates that under certain conditions, even when prices are endogenous due to simultaneity of the demand-supply system, standard panel data differencing can still recover individual demand slopes as markets become large. This follows because the source of price endogeneity when estimating individual demand is that equilibrium prices depend on individual preferences through average market demand. However, as the market grows large, the influence of any one individual's preferences on the market price becomes negligible, and we obtain the result above.
The usefulness of this result hinges on the plausibility of Assumption (ref). An important feature of Assumption (ref) is that it restricts only the time-varying component $\eta_{it}$ of the unobserved determinant $\alpha_i+\eta_{it}$ of preferences. The fixed preference component $\alpha_i$ is allowed to covary with the supply shock in any way at all. In many demand models $\eta_{it}$ is specified to be independent of all other variables, e.g. in the original Logit choice models of mcfadden1974conditional and berry1995automobile where $\eta_{it}$ has a Type I Extreme Value distribution. Such a specification is commonly used for other nonlinear panel data models as well, e.g. chamberlain1984panel. One way to motivate such a specification is to regard $\eta_{it}$ as a time-varying shock to preferences that represents individual taste for variety. Such time-varying individual preferences are generally allowed for in panel data demand models because they better fit demand data. It seems plausible that these would be unrelated to supply shocks as long as the consumer is not employed by the firms supplying the product of interest.
One potential source of covariation in changes in preferences and supply is macroeconomic shocks that are common to supply and demand. One way these can be allowed for in this model is through the use of observable regressors that depend on time. For example, the above formula can allow for the presence of a constant in the panel differences regression. The presence of a constant in the differences regression is equivalent to the presence of a time trend if several time periods are used in the regression or a time dummy in a two-period regression across different markets. The nature of Assumption (ref) in ruling out covariance between time differences in preferences and supply shocks does indicate that it is important to check for time effects in panel demand models. As such, we now extend the model to allow for a fixed time effect, where the result will hold under different but related conditions.
In this section, we extend the linear demand model of the previous section to allow for a fixed time effect. We show that bias caused by price endogeneity will disappear as the number of consumers in each market grows if individual demand and supply shocks are orthogonal across individuals and over time, and if preferences are uncorrelated across individuals and over time.
Suppose there is a good sold in many markets, each of which has $M$ consumers. We observe a sample of $N$ different consumers, $i = 1, \dots, N$, each from a different market $m$, over $T \geq 2$ time periods $t = 1,\dots,T$. In this sample we observe the quantity demanded $q_{it}$ and price $p_{it}$ of the good for each consumer $i$ in each time period $t$. Note that although we will assume in our model that prices do not vary across individuals in the same market, in the sample prices vary across individuals because each individual in the sample is drawn from a different market. This price variation is important for showing consistency for the linear demand model with a time effect. If all individuals come from the same market, in our model they face the same price and so prices would be collinear with the time effect.
Suppose that for each individual in every market we can write demand as a linear function of price,
where $\alpha_i$ is unobserved time-invariant individual heterogeneity, $\phi_t$ is fixed and unobserved individual-invariant time heterogeneity, and $\eta_{it}$ is unobserved idiosyncratic individual heterogeneity.
Let $\{i: i \in m\}$ denote the set of individuals $i$ in market $m$, which may include individuals not in the sample. Because price is common across all individuals in the same market, when analyzing a single market we will denote price as $p_{mt}$. The average quantity demanded in market $m$, where the average is taken over all individuals in market $m$, is given by
Suppose as before that in each market $m$ we can write the inverse supply model as a linear model of average supply,
where $\varepsilon_{mt}$ is an unobserved supply shock in market $m$. Solving for equilibrium prices $p_{mt}$ in market $m$ by setting demand equal to supply, we obtain
Because we assume that each individual in the sample comes from a different market, in what follows we will use $\varepsilon_{it}$ instead of $\varepsilon_{mt}$ and $\bar\eta_{it}^m$ instead of $\bar\eta_{mt}$ to denote the observation for the individual $i$ in the sample coming from market $m$.
As usual for panel data with two-way fixed effects, for each individual $i$ in the sample we can subtract off sample unit-specific means and time-specific means to eliminate the unobserved fixed effects. In particular,
where variables with a time subscript $t$ are averaged over individuals $i$ in the sample, variables with an individual subscript $i$ are averaged over time periods $t$ in the sample, and variables without any subscript are averaged over both individuals $i$ and time periods $t$ in the sample. The exact definitions of all variables are available in the Appendix.
In general $\tilde p_{it}$ will not be orthogonal to $\tilde\eta_{it}$ due to price endogeneity caused by simultaneity of the system. There are two potential sources of endogeneity. The first is that $\tilde p_{it}$ depends on $\tilde \eta_{it}^m$, which is correlated with $\tilde \eta_{it}$, as can be seen in (ref). The second, which we address with our key assumption, is that $\tilde p_{it}$ depends on $\tilde \varepsilon_{it}$, and supply shocks $\tilde \varepsilon_{it}$ may not be uncorrelated with idiosyncratic preferences $\tilde\eta_{it}$. Thus OLS on (ref) will not be consistent. However, we show that this inconsistency will disappear as the number of individuals in each market $M$ grows. This result holds when individual idiosyncratic preference shocks are uncorrelated across both individuals and time periods, and supply shocks are uncorrelated with individual idiosyncratic preferences across both individuals and time periods. As in the linear model without a time effect, the inconsistency will disappear at the rate $O(M^{-1})$.
With the additional time effect, the key condition for the inconsistency to vanish as $M$ grows is now the following one.
Assumption (ref) states that the market supply shock $\varepsilon_{it}$ is orthogonal to the individual preference shock $\eta_{js}$ and that preference shocks are uncorrelated across time and individuals. Assumption (ref) also requires that both $\eta_{it}$ and $\varepsilon_{it}$ are identically distributed over $i$ and $t$, although this assumption is for notational simplicity and can be relaxed, as discussed later.
Under this condition, we obtain the following result:
Note that because we sample individuals from independent markets, there is sufficient independent variation in $p_{it}$ across observations to make the usual formula for the probability limit of OLS valid. From this formula we see that as the number of individuals $M$ in each market grows $plim(\hat\beta)$ converges to the true $\beta$ at rate $O(M^{-1})$.
This result shows that even with a linear demand model that contains a fixed time effect, under certain conditions, the price endogeneity caused by simultaneity disappears as the number of individuals in all markets grows large. The requirements for this result to hold are stricter with the additional time effect. In particular, we now require orthogonality of the supply shock and preference shock to hold over time. Importantly, just as before, the individual preference component $\alpha_i$ of the unobserved determinant of preferences $\alpha_i + \phi_t + \eta_{it}$ is allowed to covary with all other variables, like the supply shock or the idiosyncratic preference shock. Note that Assumption (ref)(iii), which implies $Var(\eta_{it})$ is the same across all $i$ and $t$, is not crucial for the $O(M^{-1})$ rate to hold, as long as $Var(\eta_{it})$ can be uniformly upper bounded by a constant.
In order to have sufficient independent price variation we made the sampling assumption that each individual in the sample came from a different market. This sampling assumption can be relaxed to allow the sample to contain multiple individuals from the same market, so long as there remains sufficient independent price variation across individuals in the sample. We can also relax the assumption that all markets have exactly $M$ individuals; the result holds taking $M$ to be the size of the smallest market.
In this section, we extend the linear demand model to allow for a random coefficient on price. We show under similar conditions as before that the bias of the OLS slope estimator for the mean of the random coefficient will disappear as the number of consumers in the market grows.
As before, consider a single market with $M$ consumers $i=1,\dots,M$. For a sample of size $N$ of those consumers, we observe quantity $q_{it}$ and price $p_t$ of a single good for each consumer $i$ over $T \geq 2$ time periods $t = 1, \dots, T$.
Suppose we can write demand for every consumer in the market as
where $\beta_i$ is a random coefficient on price. Average demand in period $t$, where the average is taken over all individuals in the market, is
Suppose as before that we can write the inverse supply model as a linear model of average supply,
Solving for equilibrium price $p_t$ by setting demand equal to supply then gives
We will maintain the following key assumptions.
Assumption (ref) maintains similar assumptions to those imposed previously on preference and supply shocks, but additionally requires that the random coefficient be independent of both preference and supply shocks.
Our goal will be to estimate $\beta_0$, the mean of the random coefficient. We will show that the slope coefficient from the OLS regression of $q_{it}$ on $p_t$ has inconsistency that disappears with the number of consumers in the market $M$.
The additional technical assumption required for the result ensures that moments of equilibrium prices exist and are uniformly bounded. Under this additional assumption, we obtain a rate of convergence of $O(1/M)$, just as for the linear demand model.
In this section, we extend the random coefficient demand model from the previous section to allow for a fixed time effect. We show under similar conditions as before that the bias of the OLS slope estimator for the mean of the random coefficient will disappear as the number of consumers in each market grows.
As before, suppose there is a good sold in many markets, each of which has $M$ consumers. Consider a sample of $N$ individuals, $i = 1,\dots,N$, each from a different market $m$ and observed over $T \geq 2$ time periods $t = 1,\dots, T$. In the sample we observe the quantity demanded $q_{it}$ and price $p_{it}$ for each individual $i$ in each time period $t$. As before, note that while we assume prices do not vary across individuals in the same market, in the sample prices vary across individuals who come from different markets.
Suppose that for each individual in every market, demand can be written as
where $\beta_i$ is a random coefficient and $\phi_t$ is a fixed time effect. Average demand in period $t$ and market $m$, where the average is taken over the individuals in market $m$ and $p_{mt}$ denotes the price in market $m$ and period $t$, is
Suppose as before that in each market $m$ we can write the inverse supply model as a linear model of average supply,
where $\varepsilon_{mt}$ is a supply shock term. Solving for equilibrium prices $p_{mt}$ in market $m$ by setting demand equal to supply, we obtain
We will maintain the following key assumptions.
We let $\{\eta_{jt}: j \text{ in any market}, t = 1, \dots, T\}$ denote the collection of $\eta_{it}$ across all periods $t$ and all individuals $j$ in every single market, and we let $\{\varepsilon_{mt}: m \text{ is any market}, t = 1,\dots,T\}$ denote the collection of $\varepsilon_{mt}$ across all periods $t$ and all markets $m$. Assumption (ref) maintains similar assumptions to Assumption (ref) but requires stronger orthogonality of preference and supply shocks across all individuals in all markets and across all time periods.
As before, our goal will be to estimate $\beta_0$, the mean of the random coefficient. We will consider the slope estimate from the regression of $q_{it}$ on $p_{it}$ and time dummies.
The additional technical assumptions ensure that sufficient moments of prices and demeaned prices exist and are uniformly bounded.
In this paper we showed how panel data can be used to estimate individual demand models without instrumental variables, despite prices being simultaneously determined in markets. This result relies on the key assumption that idiosyncratic individual demand shocks are orthogonal to supply shocks and other individuals' idiosyncratic demand shocks, which is plausible in a wide variety of settings. We considered linear demand models and random coefficient demand models, both with and without a fixed time effect, together with a linear supply model. We characterized the rate at which bias of individual demand estimates shrinks as the number of individuals in each market grows.
An important direction for future work is extending these results from linear demand models to more general models of demand and from linear supply models to more general models of supply. We are currently working on such extensions, which rely on the intuition that individual-specific random coefficient models only require time homogeneity of individual preferences. Showing these results requires different proof techniques.
\singlespacing