Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
103,729 characters · 0 sections · 88 citation commands
Identification and Estimation of Continuous-Time Dynamic Discrete Choice Games
\@econpdfauthtrue \def#1{Jason R. Blevins}
\@econkeywordstrue \gdef{Continuous time, Markov decision processes, dynamic discrete choice, dynamic games, identification} \hypersetup{ pdfkeywords = {Continuous time, Markov decision processes, dynamic discrete choice, dynamic games, identification}}
\@econjeltrue \gdef{ C13, C35, C62, C73} \hypersetup{ pdfsubject = {JEL Classification: C13, C35, C62, C73}}
\@econabstracttrue \gdef\@abstract{This paper considers the theoretical, computational, and econometric properties of continuous time dynamic discrete choice games with stochastically sequential moves, introduced by \citet*{abbe-2016}. We consider identification of the rate of move arrivals, which was assumed to be known in previous work, as well as a generalized version with heterogeneous move arrival rates. We re-establish conditions for existence of a Markov perfect equilibrium in the generalized model and consider identification of the model primitives with only discrete time data sampled at fixed intervals. Three foundational example models are considered: a single agent renewal model, a dynamic entry and exit model, and a quality ladder model. Through these examples we examine the computational and statistical properties of estimators via Monte Carlo experiments and an empirical example using data from rust87optimal. The experiments show how parameter estimates behave when moving from continuous time data to discrete time data of decreasing frequency and the computational feasibility as the number of firms grows. The empirical example highlights the impact of allowing decision rates to vary.}
\@econacktrue \gdef{ Replication package available at \url{https://github.com/jrblevin/ctgames-qe}. I am grateful for useful comments from the editor and three referees, as well as discussions with seminar participants at Columbia University, Indiana University, Northwestern University, Stony Brook University, Texas A&M University, the University of British Columbia, the University of Chicago, the University of Iowa, the University of Michigan, and the University of Montreal, and conference attendees at the 2013 Meeting of the Midwest Econometrics Group, the 2014 Meeting of the Midwest Economics Association, the 2014 University of Calgary Empirical Microeconomics Workshop, the 2015 Econometric Society World Congress and the 2019 International Association for Applied Econometrics conference. This work builds on earlier work with Peter Arcidiacono, Patrick Bayer, and Paul Ellickson and benefited tremendously from our many discussions together. }
\hypersetup{ pdftitle={\@title} } \if@econpdfauth \hypersetup{ pdfauthor={#1} } \else \hypersetup{ pdfauthor={\@author} } \fi
\@startsection {section}{1}{\z@} {-3.5ex \@plus -1ex \@minus -.2ex} {2.3ex \@plus.2ex} {\normalbaselines\normalfont}{Introduction}
This paper studies continuous-time econometric models of dynamic discrete choice games. Work on continuous-time dynamic games by doraszelski-judd-2012, \citet*{abbe-2016} (henceforth ABBE), and others was motivated by their ability to allow researchers to compute and estimate more realistic, large-scale games and to carry out complex counterfactual policy experiments which were previously infeasible due to computational limitations.
Given the practical and conceptual benefits of continuous-time models, this paper considers identification and estimation of the rate of move arrivals in the original ABBE model, where it was assumed known. We also consider identification in a generalized model with additional heterogeneity through firm- and state-specific move arrival rates. We demonstrate these specifications via three canonical models: a single agent renewal model, a dynamic entry and exit model, and a quality ladder model. We carry out an empirical illustration using the original data of rust87optimal to estimate a continuous-time model with heterogeneous move arrival rates across states and compare with the restricted form of the original ABBE model. Based on the estimates, we conduct Monte Carlo experiments to investigate the effects of estimating the model using discrete-time data of varying frequencies. Using the quality ladder model, we conduct Monte Carlo experiments to examine the computational feasibility as the number of firms grows.
For many economic models there is no natural, fixed time interval at which agents make decisions. Despite this, it is standard practice for applied researchers to calibrate the decision interval in their empirical model to be equal to the sampling interval of their data. However, continuous-time modeling offers more flexibility by allowing agents to make decisions asynchronously at stochastic points in time. This approach eliminates simultaneous moves and introduces sequential decision-making, which better aligns with real-world scenarios in many cases and leads to significant computational advantages.
Another advantage of continuous-time modeling is the reduction of multiple equilibria by eliminating simultaneous moves. While it does not completely eliminate multiplicity, this simplifies estimation and the ability to conduct meaningful counterfactual simulations. Therefore, another benefit of allowing for heterogeneity in move arrival rates is to reduce symmetry in the model and thereby remove another source of multiplicity.
Modeling economic processes in continuous time dates back several decades to work in time series econometrics by phillips-1972, phillips-1973, sims-1971, geweke-1978, and geweke-marshall-zarkin-1986 and work on longitudinal models by heckman-singer-1986. Despite this early work on continuous-time models, discrete-time models became the de facto standard for dynamic discrete choice and now have a long, successful history in structural applied microeconometrics starting with the work of gotz80estimation, miller-1984, pakes-1986, rust87optimal, and wolpin84estimable. A recent series of papers aguirregabiria-mira-2007, bajari-benkard-levin-2007, pakes-ostrovsky-berry-2007, pesendorfer08asymptotic have shown how to extend two-step estimation techniques, originally developed by hotz93conditional and hotz94simulation, to more complex multi-agent settings. The computation of multi-agent models remains formidable, despite a growing number of methods for solving for equilibria pakes94computing, pakes01stochastic, doraszelski-satterthwaite-2010.
Dynamic decision problems are inherently complex and high-dimensional, especially in strategic games where multiple players interact. In discrete-time models, simultaneous actions introduce further complexity, as one must calculate expectations over all possible combinations of rivals' actions. This exponentially increases the number of future states to evaluate, making it infeasible to compute equilibrium in many economic environments. This has severely limited the scale and degree of heterogeneity in applied work using these methods.
doraszelski-judd-2012 showed that continuous-time models enjoy significant computational advantages. By eliminating simultaneous moves, the model ensures that state changes occur sequentially. Expectations over rival actions grow linearly with the number of players, rather than exponentially. As a result, continuous-time models significantly reduce the computational burden, allowing for faster and more scalable computation of equilibria.
ABBE demonstrated the empirical tractability of continuous-time games. They developed an econometric model retaining the aforementioned computational advantages while incorporating familiar features from discrete-time discrete choice models. They proposed a two-step conditional choice probability (CCP) estimator for their model, connecting continuous-time games with an established line of work on discrete-time dynamic games. They showed it is feasible to estimate extremely large-scale games and carry out counterfactuals in those games, which would be computationally prohibitive in a simultaneous-move, discrete-time model. ABBE illustrated these advantages through an empirical application analyzing Walmart's entry into the U.S. supermarket industry.
However, ABBE's identification results did not address identification of the rate of move arrivals. Here, we treat it as a structural parameter to identify and allow it to depend on the model state and player identity, with certain restrictions. These results are important empirically, to allow flexibility in the frequency of decisions in the model as separate from the frequency of observations in the data. As we show empirically using rust87optimal's original data, allowing and estimating varying decision rates can avoid bias in other structural parameters, such as costs, and improve our interpretation of results.
This paper builds on blevins-2017, which addressed identification of the reduced forms of continuous-time models using discrete-time data. While that work considered first-order linear systems of stochastic differential equations, we apply those results to a specific class of finite-state Markov jump processes generated by our structural model. We develop linear restrictions for our model that satisfy the conditions of Theorem 1 of blevins-2017 to identify the continuous-time reduced form of our model. We also address the question of identifying the structural primitives of our model.
Similar continuous-time models have since been used in a growing number of applications including takahashi-2015 to movie theaters, deng-mela-2018 to TV viewership and advertising, nevskaya-albuquerque-2019 to online games, agarwal-ashlagi-rees-somaini-waldinger-2021 to allocation of donor kidneys, jeziorski-2022 to the U.S. radio industry, schiraldi-smith-takahashi-2012-wp to supermarkets in the U.K., lee-roberts-sweeting-2012-wp to baseball tickets, cosman-2014-wp to bars in Chicago, mazur-2017-wp to the U.S. airline industry, kim-2021-does to the U.S. retail banking industry, and qin-vitorino-john-2022-wp to airline networks in China.
The remainder of this paper is organized as follows. In (ref), we review a generalized version of the ABBE model in which move arrival rates may vary by player and state. We establish a linear representation of the value function in terms of CCPs as well as the existence of a Markov perfect equilibrium. We then develop new identification results for the model in (ref). We use two canonical examples throughout the paper to illustrate our results: a single agent renewal model based on rust87optimal and a $2 \times 2$ entry model similar to models used by aguirregabiria-mira-2007, pesendorfer08asymptotic, and others. Finally, in (ref) we examine the computational and econometric properties via a series of Monte Carlo experiments. (ref) concludes.
\@startsection {section}{1}{\z@} {-3.5ex \@plus -1ex \@minus -.2ex} {2.3ex \@plus.2ex} {\normalbaselines\normalfont}{Continuous-Time Dynamic Discrete Choice Games}
We consider infinite horizon discrete games in continuous time indexed by $t \in [0, \infty)$ with $N$ players indexed by $i = 1, \dots, N$. We introduce a heterogeneous generalization of the ABBE model, where players may have different discount rates and move arrival rates across states. After formalizing the structural model, we establish a linear representation of the value function in terms of CCPs and existence of a Markov perfect equilibrium. We conclude the section with a comparison of discrete- and continuous-time models.
\@startsection{subsection}{2}{\z@} {-3.25ex\@plus -1ex \@minus -.2ex} {1.5ex \@plus .2ex} {\normalbaselines\normalfont}{State Space}
At any instant, the payoff-relevant market conditions can be summarized by a state vector $x \in \mathcal{X}$ with $K \equiv |\mathcal{X}| < \infty$. Each state $x \in \mathcal{X}$ contains information about the market structure (e.g., which players are active, the quality of each player) and market conditions (e.g., demographic and geographic characteristics, input prices).
States $x$ are typically $L$-dimensional vectors in $\R^L$, with components that represent player-specific states and market characteristics. For example, $x = (x_1, \dots, x_N, d)$ where $x_i$ are player-specific states (e.g., incumbency status or number of stores) and $d$ is an exogenous market characteristic (e.g., population or demand level).
Because the state space is finite there is an equivalent encoded state space representation $\mathcal{K} = \{ 1, \dots, K \}$. Although $\mathcal{X}$ is the most natural way to interpret the state, using $\mathcal{K}$ allows us to easily vectorize payoffs, value functions, and other quantities.
\begin{ex:renewal} Consider a continuous-time version of the single-agent renewal model of rust87optimal. The manager of a municipal bus company faces a dynamic decision about when to replace a bus engine. Replacement incurs an immediate cost but resets the engine's mileage, reducing maintenance costs and decreasing breakdown likelihood. A single discrete state variable $x \in \{ x_1, x_2, \dots, x_K \}$ represents the accumulated engine mileage. In our empirical example, each $x_k$ represents a mileage bin $[5000 \times (k-1), 5000 \times k)$, with $K = 90$ and a maximum of 450,000 miles. Without loss of generality, we can represent the mileage by an integer $k \in \mathcal{K} = \{ 1, \dots, K \}$. \end{ex:renewal}
\begin{ex:entry} As a second example, consider a simple model involving two firms $i \in \{ 1, 2 \}$ who sell the same good or service, each deciding whether to enter or exit a market. Each firm has two actions $j \in \{ 0, 1 \}$. The choice $j = 1$ is a switching action: enter the market if inactive, or exit the market if active, while $j = 0$ represents a continuation choice to remain active or inactive, as the case may be. Firms observe their own and each other's market activity status $x_{1k}$ and $x_{2k}$, along with an exogenous demand state $d$ that can be high ($\text{H}$) or low ($\text{L}$).
The model is a two-firm entry game with a binary exogenous state variable. The state vector $x_k$ contains $x_{1k}, x_{2k} \in \{ 0, 1 \}$ and $d_{k} \in \{ \text{L}, \text{H} \}$. The state space is
We can represent this in encoded form as $\mathcal{K} = \{ 1, 2, 3, 4, 5, 6, 7, 8 \}$. This representation will be more analytically convenient to characterize the model. \end{ex:entry}
Although our two running examples are simple, to better illustrate the ideas, in (ref) we introduce a third example: a quality ladder model of oligopoly dynamics with heterogeneous firms based on the model of ericson95markov-perfect.\footnote{As another example, ctnpl specify a continuous-time version of the dynamic entry-exit model of aguirregabiria-mira-2007.}
\@startsection{subsection}{2}{\z@} {-3.25ex\@plus -1ex \@minus -.2ex} {1.5ex \@plus .2ex} {\normalbaselines\normalfont}{Exogenous State Changes}
The state of the model can evolve over time in response to exogenous events, which we attribute to an artificial player referred to as “nature,” indexed by $i = 0$. This player is responsible for state changes that cannot be attributed to the action of any other player $i > 0$ (e.g., changes in population or per capita income). When the model is in state $k$, let $q_{kl}$ denote the hazard rate for transitions to another state $l \neq k$. The rate $q_{kl}$ may be zero if direct transitions from $k$ to $l$ are not possible, or $q_{kl}$ may be some positive but finite value representing the hazard rate of such a transition. Therefore, the overall rate at which the system leaves state $k$ for any other state $l \neq k$ is $\sum_{l \neq k}^K q_{kl}$.
\begin{ex:renewal}[continued] Suppose the exogenous mileage transition process is characterized by a rate parameter $\gamma$ governing mileage increases to the next state. This rate is constant across states for simplicity, so for all $l \neq k$ we have
\end{ex:renewal}
\begin{ex:entry}[continued] In the $2 \times 2$ entry model, there are two exogenous states: high demand ($d = \text{H}$) and low demand ($d = \text{L}$). Suppose nature switches from $\text{H}$ to $\text{L}$ at rate $\gamma_{\text{HL}}$ and back to $\text{H}$ at rate $\gamma_{\text{LH}}$. Thus, we have
\end{ex:entry}
\@startsection{subsection}{2}{\z@} {-3.25ex\@plus -1ex \@minus -.2ex} {1.5ex \@plus .2ex} {\normalbaselines\normalfont}{Decisions & Endogenous State Changes}
As in discrete time games, the players in our model can take actions that influence the evolution of the state vector. Each player has $J$ actions represented by the choice set $\mathcal{J} = \{ 0, 1, 2, \dots, J-1 \}$. When the model is in state $k$, the holding time until the next move by player $i$ is exponentially distributed with rate parameter $\lambda_{ik}$. In other words, decision times for player $i$ in state $k$ occur according to a Poisson process with rate $\lambda_{ik}$. We assume these processes are independent across players and the rates $\lambda_{ik}$ are finite for all $i$ and $k$, reflecting the fact that monitoring the state and making decisions is costly, making continuous monitoring infeasible.
In ABBE and previous applications of this framework, the rate of decisions was assumed to be known by the researcher and to be constant across players and states. For example, $\lambda_{ik} = 1$ would correspond to a decision on average once per time unit. In this paper, we consider the rates $\lambda_{ik}$ to be structural parameters to be estimated. Additional identifying restrictions will be required and therefore the specification of these rates will be important.
Let $h_{ijk}$ denote the rate at which player $i$ takes action $j$ in state $k$, with the overall decision rate in state $k$ satisfying $\sum_{j=0}^{J-1} h_{ijk} = \lambda_{ik}$. The choice-specific hazards are determined endogenously in equilibrium as discussed in detail in the following sections. When player $i$ chooses action $j$, the state jumps immediately and deterministically from $k$ to the continuation state denoted by $l(i,j,k)$.
The assumption of deterministic state changes easily accommodates decisions such as market entry, price adjustments, or construction of a new store, which are direct and certain. Our framework can also accommodate stochastic outcomes if both the decision and outcome are discrete, observable, and encoded in the state vector. The uncertainty of the outcome can be attributed to “nature” and the rates of state changes that result would be parameters of the exogenous state transition process discussed in the previous section.\footnote{Consider an example of R&D investment with an uncertain success rate. If the firm's R&D investment is an observable choice and encoded in the state vector (say, $j \in \{ 0, 1 \}$ switches the firm's R&D state $x_{i,\text{r}} \in \{ 0, 1 \}$) and if the success is observable (say, a new product is either developed or not, $x_{i,\text{p}} \in \{ 0, 1 \}$), then our model allows this by treating the new product development as an uncertain outcome determined by nature, following the R&D investment, with an estimable rate of success.}
In most economic models, the actions of players only affect their individual components of the overall state vector. For example, when a new firm enters a market it may change the firm-specific activity indicator for that firm but not the level of demand in the market. As we will discuss in more detail below, this leads to sparsity of the continuous time model and helps with identification.
\begin{ex:renewal}[continued] Since there is a single agent ($N = 1$), we drop the subscript $i$ from the notation for this example. Suppose the manager decides whether to replace a bus engine ($j=1$) or continue without replacing ($j = 0$). Hence, $\mathcal{J} = \{ 0, 1 \}$. Continuation does not change the state, but upon replacement the state resets immediately to $k = 1$, therefore
The agent makes decisions in each state $k$ at times determined by an exogenous Poisson process with rate parameter $\lambda_{k}$. This process represents the distribution of times when the manager considers whether to replace the engine of a bus in mileage state $k$. In a simple model, we may assume the decision rate is constant across states: $\lambda_{k} = \lambda$. Alternatively, we could allow that the manager evaluates buses with higher mileage more frequently than those with lower mileage:
In this case, $\lambda_{\text{L}}$ is the rate of evaluation of a low-mileage bus (in the lower half of states) and $\lambda_{\text{H}}$ is the rate at which a bus with higher mileage is monitored.
Let $h_{1k}$ denote the reduced form hazard of engine replacement in state $k$. The rate of replacement $h_{1k}$ plus the rate of continuation $h_{0k}$ in each state $k$ must be such that $h_{1k} + h_{0k} = \lambda_{k}$. Before discussing how these choice-specific hazards are determined optimally, we need to first formalize the transition dynamics of the state vector and introduce the payoff functions of the players. \end{ex:renewal}
\begin{ex:entry}[continued] In the $2 \times 2$ entry model, each firm $i$ makes decisions about entering or exiting the market in each state $k$ at rates $\lambda_{ik}$. We may believe that firms are heterogeneous, monitoring the market at different rates, but at possibly the same rate across states: $\lambda_{ik} = \lambda_i$. Alternatively, one could specify a model where firms can monitor the market more (or less) closely when demand is high ($d = \text{H}$) than when demand is low ($d = \text{L}$):
These are merely two examples. We consider a third possibility---a model where the move arrival rates depend on the endogenous decisions of the players---in (ref). \end{ex:entry}
\@startsection{subsection}{2}{\z@} {-3.25ex\@plus -1ex \@minus -.2ex} {1.5ex \@plus .2ex} {\normalbaselines\normalfont}{Payoffs}
In the continuous-time setting, we distinguish between the flow payoffs that a player receives while the model remains in state $k$, denoted $u_{ik}$, and the instantaneous choice-specific payoffs from making choice $j$ in state $k$ at a decision time $t$, denoted $c_{ijk}(t)$. The instantaneous payoffs are additively separable as $c_{ijk}(t) = \psi_{ijk} + \varepsilon_{ijk}(t)$, where $\psi_{ijk}$ is the mean payoff and $\varepsilon_{ijk}(t)$ is a choice-specific unobserved payoff. Player $i$ observes the vector $\varepsilon_{ik}(t) \equiv \left( \varepsilon_{ijk}(t), j = 0,\dots,J-1 \right)$ of choice-specific unobservables before choosing action $j$. All players and the researcher observe the state $k$, but only player $i$ observes $\varepsilon_{ik}(t)$.
\begin{ex:renewal}[continued] In the renewal model the agent faces a dynamic, stochastic cost minimization problem where the flow utility $u_{ik}$ is the flow cost of operating a bus with mileage $k$. For example, if the cost of mileage is $\beta < 0$ then a parametric flow utility function could be $u_{ik} = \beta k$. No cost is paid to continue, but a cost $\mu < 0$ is paid to replace the engine:
Following any choice $j$, the agent also receives the iid shock $\varepsilon_{ijk}$ associated with that choice. \end{ex:renewal}
\@startsection{subsection}{2}{\z@} {-3.25ex\@plus -1ex \@minus -.2ex} {1.5ex \@plus .2ex} {\normalbaselines\normalfont}{Assumptions}
Before turning to the equilibrium, we pause and collect our assumptions so far.
Assumptions (ref)--(ref) are generalized counterparts of Assumptions 1--4 of ABBE that allow for player heterogeneity and state dependent rates.\footnote{Specifically, Assumption (ref) is equivalent to Assumption 1 of ABBE, Assumptions (ref) and (ref) generalize Assumptions 2(a) and 2(b--c) of ABBE, Assumption (ref) is equivalent to Assumptions 2(d--e) of ABBE, and Assumptions (ref)--(ref) are equivalent to Assumptions 3--4 of ABBE, and Assumption (ref) generalizes Assumption 5 of ABBE.} Assumptions (ref)--(ref) were discussed above. Assumption (ref) formalizes that $j = 0$ is a costless continuation action and that all choices are observationally distinct. The first part of (ref) requires that if an inaction decision which does not change the state, denoted $j = 0$, is included in the choice set, then the instantaneous payoff associated with that choice must be zero.\footnote{The role of the choice $j = 0$ is similar to the role of the “outside good” in models of demand. Because not all agents in the market are observed to purchase one of the goods in the model, their purchase is defined to be the outside good.} This is an identifying assumption. The second part of (ref) requires actions to be meaningfully distinct in the ways they change the state.
Finally, we formalize a common distributional assumption used in applied work. We will use this assumption in examples and results throughout the paper for its tractability. This assumption implies Assumption (ref).
\@startsection{subsection}{2}{\z@} {-3.25ex\@plus -1ex \@minus -.2ex} {1.5ex \@plus .2ex} {\normalbaselines\normalfont}{Strategies and Best Responses}
A stationary Markov policy for player $i$ is a function $\delta_i: \mathcal{K} \times \R^{J} \to \mathcal{J}: (k, \varepsilon_{ik}) \mapsto \delta_i(k, \varepsilon_{ik})$ mapping each state $k$ and vector $\varepsilon_{ik}$ to an action. Associated with each policy $\delta_i$ are CCPs
Since firm $i$'s payoffs depend on rival shocks $\varepsilon_{mjk}$ only through their choices, it is sufficient to consider beliefs in terms of CCPs. Let $\varsigma_{im}$ denote player $i$'s beliefs about player $m$: a collection of $J \times K$ probabilities. Let $\varsigma_{i} = (\varsigma_{i1}, \dots, \varsigma_{i,i-1}, \varsigma_{i,i+1}, \dots \varsigma_{iN})$ denote player $i$'s beliefs about all other players. Finally, let $V_{ik}(\varsigma_i)$ denote player $i$'s expected present value in state $k$ when behaving optimally while rivals follow strategies consistent with beliefs $\varsigma_i$. The best response strategy for player $i$ is
That is, at each decision time the best response function $b_i$ assigns the action that maximizes the agent's expected payoff. The quantities on the right side are the instantaneous payoff $\psi_{ijk} + \varepsilon_{ijk}$ associated with choice $j$ plus the present discounted value of payoffs that occur in the continuation state $l(i,j,k)$ arising when player $i$ chooses action $j$ in state $k$.
\@startsection{subsection}{2}{\z@} {-3.25ex\@plus -1ex \@minus -.2ex} {1.5ex \@plus .2ex} {\normalbaselines\normalfont}{Value Function}
Given beliefs $\varsigma_i$ held by player $i$, we can define the value function (here, a $K$-vector) $V_i(\varsigma_i) = ( V_{i1}(\varsigma_i), \dots, V_{iK}(\varsigma_i) )^\top$ where the $k$-th element $V_{ik}(\varsigma_i)$ is the present discounted value of all future payoffs obtained when starting in some state $k$ and behaving optimally in future periods given beliefs $\varsigma_i$. For a small time increment $\tau$, under Assumption (ref) the probability of an event with rate $\lambda_{ik}$ occurring is $\lambda_{ik} \tau$. Given the discount rate $\rho_i$, the discount factor for such increments is $1 / (1 + \rho_i \tau)$. Thus, for small time increments $\tau$ the present discounted value of being in state $k$ is
The $o(\tau)$ term accounts for the probabilities of two or more Poisson events occurring during the small interval $\tau$, which are proportional to $\tau^2$ or smaller. Such probabilities become negligible as $\tau$ approaches zero, and thus can be ignored in the limit. Rearranging and letting $\tau \to 0$, we obtain the following recursive expression for $V_{ik}(\varsigma_{i})$:
The denominator contains the sum of the discount factor and the rates of all events that might possibly change the state. The numerator is composed of the flow payoff for being in state $k$, the rate-weighted values associated with exogenous state changes, the rate-weighted values associated with states that occur after moves by rival players, and the expected current and future value obtained when a move arrival for player $i$ occurs in state $k$. The expectation is taken with respect to the joint distribution of $\varepsilon_{ik} = ( \varepsilon_{i0k}, \dots, \varepsilon_{i,J-1,k} )^\top$.
\begin{ex:renewal}[continued] In the renewal model, the value function can be expressed very simply as follows (where the $i$ subscript and beliefs have been omitted since $N = 1$):
\end{ex:renewal}
\begin{ex:entry}[continued] In the $2 \times 2$ entry model, the value function for player $1$ in state $k$, where $x_k = (x_{k1}, x_{k2}, d_{k}) \in \{ 0, 1 \} \times \{ 0, 1 \} \times \{ \text{L}, \text{H} \}$, can be expressed recursively as (omitting beliefs $\varsigma_1$ for brevity):
where $l(0,\text{H},k)$ and $l(0,\text{L},k)$ are the continuation states when nature switches the level of demand to $\text{H}$ and $\text{L}$, respectively, when in state $k$. $\varsigma_{12jk}$ is firm 1's belief about firm 2 choosing $j$. \end{ex:entry}
\@startsection{subsection}{2}{\z@} {-3.25ex\@plus -1ex \@minus -.2ex} {1.5ex \@plus .2ex} {\normalbaselines\normalfont}{Markov Perfect Equilibrium}
Following the literature, we focus on Markov perfect equilibria. The definition requires that for each player $i$, $\delta_i^*$ is a best response in all states given the beliefs $\varsigma_i$ and that these beliefs are consistent with the strategies $\delta_m^*$ for each rival player $m$.
Following milgrom-weber-1985 and aguirregabiria-mira-2007, we characterize Markov perfect equilibria in terms of equilibrium CCPs
Henceforth, we will denote equilibrium choice probabilities and corresponding beliefs by $\sigma_{ijk}$. Thus, $\sigma = (\sigma_1, \dots, \sigma_N)$ will denote a profile of equilibrium choice probabilities and $\sigma_{-i}$ will denote the collection of rival choice probabilities that constitute player $i$'s beliefs.
ABBE proved that such an equilibrium exists when players share common move arrival and discount rates and when the move arrival rates do not vary across states (i.e., $\lambda_{ik} = \lambda$ and $\rho_i = \rho$ for all $i$ and $k$). The following theorem extends this to the present generalized model with heterogeneity. The proof, and all others, appears in (ref).
\@startsection{subsection}{2}{\z@} {-3.25ex\@plus -1ex \@minus -.2ex} {1.5ex \@plus .2ex} {\normalbaselines\normalfont}{Linear Representation of the Value Function}
Before proceeding, we revisit one of the central results of ABBE (Proposition 2), a continuous-time analog of hotz93conditional for discrete-time models. Restated below as Lemma (ref), ABBE showed that differences in choice-specific value functions---that is, $[\psi_{ijk} + V_{i,l(i,j,k)}(\sigma)] - [\psi_{ij'k} + V_{i,l(i,j',k)}(\sigma)]$ for two choices $j$ and $j'$---are identified directly as functions of the CCPs $\sigma_i$.
This result will prove useful for vectorizing the value function. Let $\Sigma_{m}(\sigma_{m})$ denote the transition matrix implied by the CCPs $\sigma_{m}$ and the continuation state function $l(m,\cdot,\cdot)$. That is, the $(k,l)$ element of the matrix $\Sigma_{m}(\sigma_{m})$ is the probability of transitioning from state $k$ to state $l$ as a result of an action by player $m$ given the CCPs $\sigma_{m}$. Let $Q_0 = (q_{kl})$ denote the intensity matrix for exogenous state transitions and let $\tilde{Q}_0 = Q_0 - \operatorname{diag}(q_{11}, \dots, q_{KK})$ be the matrix formed by taking $Q_0$ and setting the diagonal elements to zero.
With this notation and Lemma (ref) in hand, following (ref), for given equilibrium CCPs $\sigma$ we define the operator $\Gamma_i^{\sigma}$ as
where $D_i$ is the $K \times K$ diagonal matrix with elements $(D_i)_{kk} = 1/(\rho_i + \sum_{l \neq k} q_{kl} + \sum_{m=1}^N \lambda_{mk})$, $L_m = \operatorname{diag}(\lambda_{m1}, \dots, \lambda_{mK})$ is a diagonal matrix containing the move arrival rates for player $m$, $C_i(\sigma_{i})$ is the $K \times 1$ vector containing the ex-ante expected value of the instantaneous payoff $c_{ijk} = \psi_{ijk} + \varepsilon_{ijk}$ for player $i$ in each state $k$ given the best response probabilities $\sigma_i$. That is, $k$-th element of $C_i(\sigma_i)$ is $\sum_{j=0}^{J-1} \sigma_{ijk} \left[ \psi_{ijk} + e_{ijk}(\sigma_i) \right]$, where $e_{ijk}(\sigma_i)$ is the expected value of $\varepsilon_{ijk}$ given that action $j$ is chosen:
By Lemma (ref), the choice-specific value differences on the right-hand side are in turn functions of player $i$'s CCPs $\sigma_i$. Hence, holding fixed the equilibrium beliefs $\sigma$, the corresponding value function is a fixed point of $\Gamma_i^{\sigma}$: $V_i = \Gamma_i^{\sigma}(V_i)$.
Collecting terms involving $V_i$ in (ref) and solving leads to a linear representation of the value function in terms of CCPs, rate parameters, and payoffs as formalized in the following Theorem. This representation generalizes Proposition 6 of ABBE and forms the basis of the identification result later in Section (ref). It is analogous to a similar result for discrete time games by pesendorfer08asymptotic.
\@startsection{subsection}{2}{\z@} {-3.25ex\@plus -1ex \@minus -.2ex} {1.5ex \@plus .2ex} {\normalbaselines\normalfont}{Continuous Time Markov Jump Processes Representation}
The model's reduced form is a finite state Markov jump process, a stochastic process $X(t)$ indexed by $t \in [0, \infty)$ taking values in a finite state space $\mathcal{X} = \lbrace 1, \dots, K \rbrace$. If we observe this process at time $t$ and state $X(t)$, it remains in this state for a random duration $\tau$ before transitioning to another state $X(t + \tau)$. The duration $\tau$ is the holding time. A trajectory of this process is a piecewise-constant, right-continuous function of time. Jumps occur according to a Poisson process and holding times between jumps are exponentially distributed. For fundamental properties of Markov jump processes, see karlin75first or chung-1967.
A finite Markov jump process can be summarized by its intensity matrix or infinitesimal generator matrix. Consider the intensity matrix for nature, $Q_0 = (q_{kl})$ where for $k \neq l$ $q_{kl} = \lim_{h \to 0} \frac{\Pr\left[X(t+h) = l \mid X(t) = k \right]}{h}$ is the probability per unit of time that the system transitions from state $k$ to $l$ and the diagonal elements are $q_{kk} = -\sum_{l \neq k} q_{kl}$ so that the row sums equal zero. Holding times before leaving state $k$ follow an exponential distribution with rate parameter $-q_{kk}$. Conditional on leaving state $k$, the system transitions to state $l \neq k$ with probability $q_{kl} / \sum_{l \neq k} q_{kl} = -q_{kl} / q_{kk}$.
For discrete time data, the exact times when actions and state changes occur are unobserved. With equispaced data (e.g., annual or quarterly) only the states at the beginning and end of each period of length $\Delta$ are observed. Although we cannot know the exact sequence of actions and state changes, the model allows us to determine the likelihood of any transition occurring over a time interval of length $\Delta$ using the transition matrix, denoted $P(\Delta)$.
Let $P_{kl}(\Delta) = \Pr\left[X(t+\Delta) = l \mid X(t) = k\right]$ denote the probability that the system is in state $l$ after a period of length $\Delta$ given that it was initially in state $k$. The transition matrix $P(\Delta) = (P_{kl}(\Delta))$ is the corresponding $K \times K$ matrix of these probabilities and is given by the matrix exponential of the intensity matrix $Q$ scaled by the time interval $\Delta$:
This is the matrix analog of the scalar exponential $\exp(x)$ for $x \in \R$.\footnote{Although we cannot calculate the infinite sum (ref) exactly, we can compute $\exp(\Delta Q)$ numerically using known algorithms implemented in the Fortran package Expokit sidje98expokit or the expm command in Matlab. See sherlock21direct for recent discussion of the uniformization method.}
In the dynamic games we consider, the state space dynamics can be fully characterized by $N + 1$ competing Markov jump processes with intensity matrices $Q_0, Q_1, \dots, Q_N$ for nature and each player. The aggregate intensity matrix is defined as $Q \equiv Q_0 + Q_1 + \dots + Q_N$.
\begin{ex:renewal}[continued] Consider the $Q$ matrix implied by the continuous-time single-agent renewal model. The state variable in the model is the total accumulated mileage of a bus engine, $\mathcal{K} = \{ 1, \dots, K \}$. The exogenous state transition process is characterized by a $K \times K$ intensity matrix $Q_0$ on $\mathcal{K}$ with one parameter, $\gamma$, governing the rate of mileage increases:
The intensity matrix for state changes induced by the agent is
The aggregate intensity matrix in this case is $Q = Q_0 + Q_1$:
\end{ex:renewal}
\begin{ex:entry}[continued] Let $h_{ik}$ be the hazard of player $i$ switching from active to inactive or vice versa in state $k$. We have dropped the $j$ subscript here for notational simplicity since $j=0$ does not change the state. Let $\gamma_{\text{LH}}$ and $\gamma_{\text{HL}}$ be the rates at which nature switches between demand states (i.e., demand moves from low to high at rate $\gamma_{\text{LH}}$). The aggregate state space dynamics are illustrated in (ref). Recall that the reduced form hazards $h_{ik}$ of firm $i$ taking action $j=1$ in state $k$ are related to the structural quantities through the relation $h_{ik} = \lambda_{ik} \sigma_{ijk}$.
The state transition hazards can be characterized by an $8 \times 8$ intensity matrix $Q$. Note that firms cannot change the demand state, firms cannot change each other's states, and nature cannot change the firms' states. Therefore, the overall intensity matrix is a sparse matrix of the form
The diagonal elements have been omitted for brevity. Importantly, the locations of the nonzero elements are distinct because the state-to-state communication patterns differ. Therefore, given $Q$ we can immediately determine $Q_0$, $Q_1$, and $Q_2$. \end{ex:entry}
\@startsection{subsection}{2}{\z@} {-3.25ex\@plus -1ex \@minus -.2ex} {1.5ex \@plus .2ex} {\normalbaselines\normalfont}{Comparison with Discrete Time Models}
We conclude with remarks on continuous time and discrete time models. First, consider a typical discrete time model where agents move simultaneously and the period between decisions is calibrated to the sampling period of the data. In an entry/exit setting where the choice set is $\mathcal{J} = \{ 0, 1 \}$, there is exactly one entry or exit decision per year. In discrete time data, passive continuation actions are coded as active decisions, but in reality they represent the absence of an active choice during the period. Consider a chain store setting where the choice is the net number of stores to open during the year, $\mathcal{J} = \{ -J, \dots, J \}$. This implies at most $J$ openings or closings per year. Hence, $J$ must be chosen to be the maximum number of possible stores opened or closed by any chain firm in any period.
Now consider a continuous time model with a common move arrival rate $\lambda$ for all players and states. In the entry/exit setting, the choice set is still $\mathcal{J} = \{ 0, 1 \}$ which implies on average $1/\lambda$ entries or exits per year. Multiple entries and exits are allowed, and the parameters imply a distribution over the number and type of such events. The choice set represents possible instantaneous state changes, so in the chain store expansion example, if no more than one store opens or closes simultaneously, we specify $\mathcal{J} = \{ -1, 0, 1 \}$. This implies on average at most $1 / \lambda$ openings or closings per year. In our continuous time model the rate $\lambda$ is a free parameter that can adjust to match the data, not imposing an ad hoc restriction on the number of actions per unit of time. The time-aggregated implications of the continuous time model are not functionally different if we change the time period and are unrelated to the sampling period, a feature of the data collection process.
\@startsection {section}{1}{\z@} {-3.5ex \@plus -1ex \@minus -.2ex} {2.3ex \@plus.2ex} {\normalbaselines\normalfont}{Identification}
Due to time aggregation, our identification analysis separates the data issue---that we may only observe $P(\Delta)$ instead of $Q$---from recovering the structural parameter $\theta$ from the continuous-time reduced form $Q$. We proceed in two steps; researchers with continuous time data can begin with the second step.\footnote{Although we take a sequential approach to identification, a direct approach from $P(\Delta)$ to $\theta$ may also be possible. While a sequential approach could, in principle, impose additional constraints when identifying $Q$ from $P(\Delta)$ that are unnecessary for directly identifying $\theta$ from $P(\Delta)$, in this case we exploit non-parametric structural restrictions from the model to identify $Q$ in the first step, mitigating this concern.} Deriving the structural model's implications is a bottom-up exercise: the structural primitives $u$ and $\psi$ imply value functions $V$ which imply choice probabilities $\sigma$. These probabilities with move rates $\lambda$ and state transitions by nature $Q_0$ imply an intensity matrix $Q$. Given the $Q$ matrix and a sampling process, this implies a data generating process. For a fixed sampling interval $\Delta$ the distribution of observable data is $P(\Delta) = \exp(\Delta Q)$.
The identification problem requires us to consider the inverse problem. These steps are represented in (ref). If the complete continuous time process is observable, then $Q$ is trivially identified and we can move to identification of the structural model. However, for discrete time data we must use our knowledge of the data generating process, represented by the transition matrix $P(\Delta)$ for an interval $\Delta$, to derive conditions under which we can uniquely determine the reduced form intensity matrix $Q$. We show this is possible under mild conditions using restrictions that the structural model places on the $Q$ matrix.
With $Q$ in hand, we turn to the structural primitives of the model: the flow payoffs $u$ and instantaneous payoffs $\psi$. We show that knowledge of $Q$ allows us to recover these structural primitives with a smaller number of additional identifying restrictions than are required in discrete time models. This is due to the absence of simultaneous moves at any given instant, which is also the source of the computational efficiency of the model.
\@startsection{subsection}{2}{\z@} {-3.25ex\@plus -1ex \@minus -.2ex} {1.5ex \@plus .2ex} {\normalbaselines\normalfont}{Identification of $Q$}
With continuous-time data, identification and estimation of the intensity matrix for finite-state Markov jump processes is well-established billingsley61statistical. However, when a continuous-time process is only sampled at discrete points in time, the parameters of the underlying continuous-time model may not be point identified.\footnote{This is known as the aliasing problem and has been studied in the context of continuous-time systems of stochastic differential equations sims-1971, phillips-1973, hansen-sargent-1983, hansen-sargent-1991-ch9, geweke-1978, kessler-rahbek-2004, mccrorie-2003, blevins-2017. See Figure 1 of blevins-2017 for an illustration in the frequency domain, where the problem is perhaps most obvious.} In the present model, the concern is that there may be multiple $Q$ matrices which give rise to the same data generating process, which is the transition probability matrix $P(\Delta)$ in the case of fixed sampling at an interval $\Delta$. \footnote{A related issue is the embeddability problem: could the transition matrix $P(\Delta)$ have been generated by a continuous-time Markov jump process for some intensity matrix $Q$ or some discrete-time chain over fixed time periods of length $\delta$? This problem was first proposed by elfving37zur. kingman62imbedding derived the set of embeddable processes with $K = 2$ and johansen74some gave an explicit description of the set for $K = 3$. singer-spilerman-1976 summarize several known necessary conditions for embeddability involving testable conditions on the determinant and eigenvalues of $P(\Delta)$. We assume throughout that the continuous time model is well-specified and that such an intensity matrix exists.}
In discrete time settings, a similar identification problem is masked when assuming the unknown frequency of moves equals the (known) sampling frequency hong15aggregated. Suppose agents move at intervals of length $\delta$ with transition matrix $P_0$ while the data sampling interval is $\Delta > \delta$. Then the mapping between the data (equispaced observations at length $\Delta$) and the transition matrix is: $P(\Delta) = P_0^{\Delta/\delta}$. Generally, there are multiple solutions to this equation gantmacher-1959, singer-spilerman-1976, meaning identification of $P_0$ is non-trivial.
Previous work on this identification problem seeks conditions on the observable discrete-time transition matrix $P(\Delta)$. We briefly review these results in the next subsection, but our approach is to show that one can identify $Q$ via identifying restrictions on the primitives of the underlying structural model and that such restrictions arise from the model itself. These can be viewed as exclusion restrictions.
For example, in applications there are typically player-specific components of the state vector where player $i$ cannot change the player-specific state of player $j$ and vice-versa. In an entry-exit model, such a state is incumbency status: players can enter and exit by their own action, but no player can enter or exit on behalf of another. Similarly, if the state vector has components that are exogenous state variables, such as population, then any state changes involving those variables must be due to nature and not by any player. This natural structure implies many linear restrictions on the $Q$ matrix. We show that restrictions of this form serve to limit the domain of the mapping $Q \mapsto \exp(\Delta Q) = P(\Delta)$ to guarantee the intensity matrix $Q$ is identified.
\@startsection{subsubsection}{3}{\z@} {-3.25ex\@plus -1ex \@minus -.2ex} {1.5ex \@plus .2ex} {\normalbaselines \normalfont}{Identification of Unrestricted $Q$ Matrices}
Returning to identification of $Q$, recall that the question is whether there exists a unique matrix $Q$ that leads to the observed transition matrix $P(\Delta) = \exp(\Delta Q)$ when observations are sampled at intervals of length $\Delta$. The matrix logarithm $\ln P(\Delta)$ is not unique in general gantmacher-1959, singer-spilerman-1976, so the question amounts to finding conditions for a unique solution.
Previous, mathematical treatments view the relationship $\exp(\Delta Q) = P(\Delta)$ from the perspective of the transition matrix $P(\Delta)$. In such cases there is no underlying model that generates $Q$, so $Q$ itself is the primitive of interest and is unrestricted (subject to being a valid intensity matrix). Most previous work focused on finding sufficient conditions on the matrix $P(\Delta)$ to guarantee that $\ln P(\Delta)$ is unique. For example, if the eigenvalues of $P(\Delta)$ are distinct, real, and positive, then $Q$ is identified culver-1966. More generally, culver-1966 proved that $Q$ is identified if the eigenvalues of $P(\Delta)$ are positive and no elementary divisor (Jordan block) of $P(\Delta)$ appears more than once. Other sufficient conditions include $\min_k\{ P_{kk}(\Delta) \} > 1/2$ cuthbert-1972 and $\det P(\Delta) > \operatorname{\mathrm{e}}^{-\pi}$ cuthbert-1973. See singer-spilerman-1976 for a summary.
Other conditions involve alternative sampling schemes. For example, $Q$ is identified if the sampling $\Delta$ is sufficiently small cuthbert-1973, singer-spilerman-1976, hansen-sargent-1983. Alternatively, $Q$ is identified if the process is sampled at distinct intervals $\Delta_1$ and $\Delta_2$ where $\Delta_2 \neq k \Delta_1$ for any integer $k$ singer-spilerman-1976.
The first conditions---restrictions on $P(\Delta)$---are based on a “top down” approach and are undesirable when $Q$ is generated by an underlying model. The second conditions are based on how the continuous time process is sampled, which cannot be changed if data are already collected at regular intervals. Instead, we take a “bottom up” approach which allows economic theory to inform our identification conditions via restrictions on $Q$ that guarantee uniqueness of $\ln P(\Delta)$. More compelling conditions involve cross-row and cross-column restrictions on the $Q$ matrix and known zeros of the $Q$ matrix. Such restrictions arise naturally once players, actions, and resulting state transitions are defined.
\@startsection{subsubsection}{3}{\z@} {-3.25ex\@plus -1ex \@minus -.2ex} {1.5ex \@plus .2ex} {\normalbaselines \normalfont}{Structural Restrictions for Identification of $Q$}
The problem of identifying continuous time models with only discrete time data has appeared previously in the econometrics literature, in work by phillips-1973 on continuous time regression models. He considered multivariate, continuous-time, time-homogeneous regression models of the form $y'(t) = A y(t) + \xi(t)$, where $y(t)$ is an $n \times 1$ vector and $A$ is an $n \times n$ structural matrix. He discussed the role of prior information on $A$ and how it can lead to identification. He showed that $A$ is identified given discrete time observations on $y$ if $A$ satisfies certain rank conditions.
Our identification strategy is inspired by this work, but our model differs because the $Q$ matrix is an intensity matrix (rather than an arbitrary matrix of regression coefficients) and has sparse structure dictated by an underlying structural model. Yet, there are similarities: the present model can be characterized by a system of differential equations, where the intensity matrix $Q$ plays a role similar to the matrix $A$. If $Q$ is a valid intensity matrix, then the functions $P(\Delta)$ solving this system are the transition matrices of continuous-time stationary Markov chains chung-1967.
The structural model restricts $Q$ to a lower-dimensional subspace since it is sparse and must satisfy both within-row and across-row restrictions, and given the results above it seems likely that these restrictions could lead to identification of $Q$. That is, even if there are multiple matrix solutions to the equation $P(\Delta) = \exp(\Delta Q)$, it is unlikely that two of them simultaneously satisfy the restrictions of the structural model. We return to the two examples introduced previously to illustrate this idea.
\begin{ex:renewal}[continued] In the single-agent renewal model the aggregate intensity matrix is given in (ref). The number of nonzero hazards is substantially less than the total. Consider $K = 90$: there are $90^2 - 90 = 8,010$ non-trivial state-to-state transitions. Only 178 are permitted at any instant: 89 due to nature and 89 by player action. The remaining 7,832 transitions are not possible in a single event. Nature cannot decrease mileage and can only increase it by one state at a given instant (although multiple state jumps are possible over an interval). The agent can only reset mileage to the initial state. Therefore, there are 7,832 known zeros of $Q$---more than sufficient to uniquely identify $Q$. Given $Q$, we can separately determine both $Q_0$ and $Q_1$. The choice-specific hazards $h_{1k}$ are products of the move arrival rates and conditional choice probabilities, which introduces shape restrictions on $h_{1k} = \lambda_k\sigma_{1k}$ across states $k$. \end{ex:renewal}
\begin{ex:entry}[continued] In the $2 \times 2 \times 2$ entry example, the aggregate intensity matrix is given by (ref). Some transitions cannot happen at all, such as $(0,1,\text{L})$ to $(1,0,\text{L})$. The remaining transitions can happen only due to the action of one of the firms, but not the other. For example, moving from $(0,0,\text{H})$ to $(1,0,\text{H})$ is only possible if firm $1$ chooses to become active. From any state, the set of other states to which either firm can move the state as a result of an action is limited naturally by the model and the definition of the state space. This structure yields intensity matrices that are sparse, which makes identification of $Q$ more likely even with time aggregation since any observationally equivalent $Q$ matrix must have the same sparsity pattern. Finally, given $Q$ we can again separately recover $Q_0$, $Q_1$, and $Q_2$. \end{ex:entry}
Similar sparse structures arise in even models with large numbers of players and millions of states, as in the application of ABBE. In light of this lower-dimensional structure, we build on the results of blevins-2017 who gave sufficient conditions for identification in first-order linear systems of stochastic differential equations. We apply those results to the case of finite-state Markov jump processes generated by our structural model.
The key insight is that structural restrictions on $Q$ of the form $R \operatorname{vec}(Q) = r$ can rule out alternative solutions to the matrix exponential equation $\exp(\Delta \tilde{Q}) = P(\Delta)$. When there are sufficiently many linearly independent restrictions, the intensity matrix $Q$ is uniquely identified. Adapting Theorem 1 of blevins-2017 to finite-state Markov jump processes, we require at least $\left\lfloor\frac{K-1}{2}\right\rfloor$ linear restrictions when $R$ has full rank.
The following theorem establishes that there are sufficiently many full rank restrictions to identify $Q$ in a broad class of games. This theorem includes exogenous market-specific state variables and shows that such states increase the number of zero restrictions and make identification of $Q$ more likely, as do player-specific state variables.
The sparsity of $Q$ helps and is increasing in both the number of exogenous states $K_0$ and player-specific states $K_1$, but decreasing in the number of choices $J$. Therefore, for identification we need either a sufficiently large number of states or a sufficiently small number of choices. Fortunately, in most applications $J$ is small relative to $K$---particularly in continuous time models as discussed in Section (ref).
\begin{ex:entry}[continued] Our running entry model example is a binary choice game with $N = 2$, $J = 2$, $K_0 = 2$, and $K_1 = 2$, so by (ref) $Q$ is identified. \end{ex:entry}
Furthermore, we can see that any binary choice game ($N > 1$ with $J = 2$) with meaningful player-specific states ($K_1 > 1$) is identified, regardless of the number of players or exogenous market states $K_0$. The sufficient condition in this case simplifies to $K_0 (K_1^N - 1) \geq N - \frac{1}{2}$. When $K_0 \geq 1$ and $K_1 \geq 2$ we have $K_0 (K_1^N - 1) \geq 2^N - 1$ which exceeds $N - \frac{1}{2}$ for integers $N > 1$.
\@startsection{subsubsection}{3}{\z@} {-3.25ex\@plus -1ex \@minus -.2ex} {1.5ex \@plus .2ex} {\normalbaselines \normalfont}{Identification of $Q_i$}
Once the $Q$ matrix is known---or in the case of continuous-time data, identified directly---we need to ensure that in any particular state it does not represent a mixture over potentially multiple equilibria. To guarantee this, we invoke an assumption corresponding to Assumption 6 of ABBE, which was in turn a continuous-time version a similar assumption required for identification and estimation of discrete time dynamic games bajari-benkard-levin-2007, aguirregabiria-mira-2007. See aguirregabiria-mira-2010 for a survey.
In a model with a unique equilibrium---for example the single agent renewal model---this assumption is satisfied trivially. In games, it requires that in any markets where the game is in the same state, the same equilibrium is played. We need this assumption no matter if we observe discrete time data from $P(\Delta)$, generated from some continuous-time $Q$, or we observe continuous-time data generated from $Q$ directly.
Next, we make the following assumption which requires that given the aggregate intensity matrix $Q$, we can determine the player-specific intensity matrices $Q_i$.
Assumption (ref) can be easily verified by inspecting $Q$ in both running examples since players cannot change each other's state variables and actions by nature can be distinguished from player actions. Note that the diagonal elements are unimportant: if the off-diagonal elements of each $Q_i$ can be identified from $Q$, then diagonal elements equal the negative sum of the off-diagonal elements. In the renewal example $Q$ is given in (ref) and for the two-player entry model in (ref). A sufficient condition for (ref) is that continuation states resulting from actions of different players are distinct: for all players $i$ and $m \neq i$ and all states $k$,
\@startsection{subsection}{2}{\z@} {-3.25ex\@plus -1ex \@minus -.2ex} {1.5ex \@plus .2ex} {\normalbaselines\normalfont}{Identification of Hazards, Value Functions and Payoffs}
We now establish that the value functions, instantaneous payoffs, and utility functions are identified. Let $V_i = ( V_{i1}, \dots, V_{iK} )^\top$ denote the $K$-vector of valuations for player $i$ in each state. Let $\psi_{ij} = ( \psi_{ij1}, \dots, \psi_{ijK} )^\top$ denote the $K$-vector of instantaneous payoffs for player $i$ making choice $j$ in each state and let $\psi_i = (\psi_{i1}^\top, \dots, \psi_{i,J-1}^\top)^\top$.
Importantly, we note that when $j = 0$ is a latent or unobserved continuation action, it is not possible to identify the rates $h_{i0k}$ even with continuous time data, so we cannot immediately treat them as identified quantities. Under Assumption (ref), the relationship between identified hazards $h_{ijk}$ for $j > 0$ and the unknown quantities $h_{i0k}$, $\psi_{ijk}$, and $V_{ik}$ takes the form of a log-linear system. Let $h_i^+$ denote the vector of identified hazards for choices $j > 0$ and $h_i^0 = (h_{i01}, \dots, h_{i0K})^\top$ denote the vector of unidentified hazards for the continuation action $j = 0$. The hazards for choices $j = 1, \dots, J-1$ are identified from $Q$, but without additional restrictions, the system has $2K$ more unknowns than equations, preventing identification of the remaining unknowns.
The following theorem shows that with $2K$ appropriate linear restrictions, all these quantities can be identified. Notably, the number of restrictions required per player is independent of the number of players in the game, so the total number of identifying restrictions is only linear in $N$. This contrasts with discrete time models where the number of restrictions needed is exponential in $N$ pesendorfer08asymptotic.
It is helpful now to consider some examples. If we assume that the instantaneous payoffs are constant across $k$, as in many applications of dynamic games, this implies $\psi_{ijk} - \psi_{ijl} = 0$ for all choices $j > 0$ and all states $l \neq k$, giving $(J-1)(K-1)$ restrictions per player. When $J=2$, we still need $K+1$ additional restrictions. If we also assume the move arrival rate is constant across state ($\sum_{j=0}^{J-1} h_{ijk} = \sum_{j=0}^{J-1} h_{ijl}$ for all $l \neq k$), we have $K-1$ restrictions. Then even if $J=2$, only $2$ additional restrictions are needed.
Additional full-rank restrictions are possible for certain applications. Examples include states where the value function is known, e.g., if $V_{ik} = 0$ when a firm has permanently exited. Exclusion restrictions of the form $V_{ik} = V_{ik'}$ are possible, where $k$ and $k'$ are two states that differ only by a rival-specific state and are payoff equivalent to firm $i$. In all of these cases, the rank condition can be verified by inspection. We also note that Theorem (ref) does not consider identification restrictions across players, but in practice these can provide additional identifying restrictions.
\begin{ex:renewal}[continued] In the single-agent renewal model, since the replacement cost does not depend on the mileage state we have $\psi_{1k} = \mu$ for all $k$. This yields $K-1$ restrictions of full rank of the form $\psi_{1k} - \psi_{11} = 0$ for $k = 2, \dots, K$. If we also assume the rate of move arrivals is constant across two subsets of states (i.e., $\lambda_{\text{L}}$ and $\lambda_{\text{H}}$), this yields $K-2$ additional restrictions. The linearity of the utility function also imposes restrictions on $V$, and although this does not fit in the linear restriction framework of (ref) it also contributes to identification of $\psi$ and $V$. \end{ex:renewal}
\begin{ex:entry}[continued] In the simple two-player entry-exit model, we may suppose that the entry costs and scrap values are independent of the market state (high or low demand) and whether a rival is present. In other words, $\psi_{i1k} - \psi_{i11} = 0$ for all states $k$, yielding $K-1$ restrictions per player. Additionally, if we assume the rate of move arrivals is firm-specific ($\lambda_{ik} = \lambda_i$), this yields $K-1$ restrictions per player. Alternatively, we considered rates depending only on the level of demand ($\lambda_{\text{L}}$ and $\lambda_{\text{H}}$). This specification would yield $K-2$ restrictions. \end{ex:entry}
Finally, we note that in practice the overall rate of actions can be identified through the nonlinear restrictions imposed by the distributional assumptions on the error term, which imply shape restrictions on the choice probabilities across states. These are difficult to characterize in the linear restriction framework we have used here, but in practice parametric assumptions will aid identification in addition to the linear restrictions considered above.
\@startsection{subsection}{2}{\z@} {-3.25ex\@plus -1ex \@minus -.2ex} {1.5ex \@plus .2ex} {\normalbaselines\normalfont}{Identification of the Payoffs}
Having established the identification of value functions, instantaneous payoffs, and hazards, we now turn to the identification of flow payoffs. The following theorem shows that these payoffs can be recovered from the previously identified quantities through the linear representation of the value function in (ref) established by Theorem (ref).
\@startsection {section}{1}{\z@} {-3.5ex \@plus -1ex \@minus -.2ex} {2.3ex \@plus.2ex} {\normalbaselines\normalfont}{Empirical Example and Monte Carlo Experiments}
In this section, we describe an empirical example along with a series of Monte Carlo experiments conducted using the single-agent renewal model. Additional experiments using a dynamic quality ladder model are presented in (ref).
\@startsection{subsection}{2}{\z@} {-3.25ex\@plus -1ex \@minus -.2ex} {1.5ex \@plus .2ex} {\normalbaselines\normalfont}{Maximum Likelihood Estimation}
The model can be estimated using maximum likelihood if equilibria can be enumerated or there is a unique equilibrium. Since this paper focuses on identification, rather than developing a new estimator, our Monte Carlo experiments use the maximum likelihood estimator with value function iteration.\footnote{More generally, methods proposed for discrete time models, such as the homotopy method borkovsky10homotopy, besanko10learning, bajari-hong-krainer-nekipelov-2010-wp or recursive lexicographical search iskhakov16recursive, could possibly be adapted to our model, but this is beyond the present scope.} Multiplicity of equilibria is not a concern for the single agent model and appears not to be an issue in practice for the continuous-time oligopoly model specifications considered in the appendix, although we have not proven that there is a unique equilibrium in the latter case.
However, in models with multiple equilibria, this maximum likelihood procedure, which relies on value function iteration, is potentially unstable. In such cases, we recommend two-step estimators that avoid this issue. ABBE introduced a two-step PML (pseudo maximum likelihood) estimator, similar to the CCP estimator of hotz93conditional for discrete-time single-agent models. More recently, ctnpl developed the continuous time NPL (CTNPL) estimator, an iterative estimator following aguirregabiria-mira-2007. However, these two-step methods assume the rate of move arrivals $\lambda$ is known. Adapting them to the general case remains an open question for future research. Similarly, other estimators for discrete time models such as those by aguirregabiria-marcoux-2019 and dearing-blevins-2024 could be adapted to the current framework.
We focus on the maximum likelihood estimator for the empirical example and simulations. This allows us to examine the computational properties and how estimates behave when the sampling frequency of the data changes without two-step estimation error.
With continuous-time data, we have a sample of $\bar{N}$ tuples $(\tau_n,i_n,a_n,k_n,k_n')$. Each describes a jump or move where: $\tau_n$ is the holding time since the previous event, $i_n$ is the player index ($i_n = 0$ is nature), $a_n$ is the action taken by player $i_n$, $k_n$ denotes the state at the time of the event, and $k_n'$ denotes the state immediately after. Let $g(\tau; \lambda)$ and $G(\tau; \lambda)$ denote the pdf and cdf of $\operatorname{\mathrm{Expo}}(\lambda)$. Let $\ell_n(\theta)$ denote the likelihood of observation $n$:
Here, $q(k,k';\theta)$ denotes the absolute value of the $(k,k')$ element of $Q(\theta)$ for given parameters $\theta$. $q_0(k,k';\theta)$ and $q_N(k,k';\theta)$ similarly denote elements of $Q_0$ and $\sum_{i = 1}^N Q_i$. $p(k,k';\theta)$ denotes the probability of a jump from $k$ to $k'$ conditional on a jump occurring. The full log-likelihood of the sample of $\bar{N}$ observations on $[0,T]$ is
The final term is the probability of not observing an event on $(t_{\bar{N}}, T]$.
With discrete-time data sampled at equispaced intervals $\Delta$ our sample consists of a collection of states $\lbrace k_{1}, \dots, k_{\bar{N}} \rbrace$ with $\bar{N}$ observations. The likelihood function is given by:
where $P(k,l; \Delta, \theta)$ denotes the $(k,l)$ element of the transition matrix induced by $\theta$.
\@startsection{subsection}{2}{\z@} {-3.25ex\@plus -1ex \@minus -.2ex} {1.5ex \@plus .2ex} {\normalbaselines\normalfont}{Single Agent Renewal Model: Empirical Results}
We consider the single-agent binary choice (bus engine replacement) model. We first estimate a continuous time version of the model using the same data rust87optimal used to estimate the original discrete time model. We then use the estimates to calibrate parameters for Monte Carlo experiments.
The model allows for heterogeneity in move arrival rates across states. We begin with a simpler model with a constant rate, $\lambda_k = 1$ for all $k$, as in ABBE. Next, we allow $\lambda$ to vary freely and estimate it. Finally, we estimate the model with heterogeneous decision rates where $\lambda_{\text{L}}$ is the rate for buses with mileage states $k = 1, 2, \dots, \left\lfloorK / 2\right\rfloor$ and $\lambda_{\text{H}}$ is the rate for mileage states $k = \left\lfloorK / 2\right\rfloor + 1, \dots, K$. The parameters to estimate are $\theta = (\lambda_{\text{L}}, \lambda_{\text{H}}, \gamma, \beta, \mu)$, which include the move arrival rates, the rate of mileage increase $\gamma$, the mileage cost parameter $\beta$, and the engine replacement cost $\mu$.
The dataset consists of monthly bus mileage recordings and the months of bus engine replacement. We provide statistics on the number of observations per bus group in Table (ref). The smallest time horizon per bus was 24 months (2 years) for group 1. The longest was 125 months ($\approx{}10$ years) for groups 5--9. As reported by rust87optimal, engine replacement occurred on average after 5 years (60 months) at over 200,000 miles. This provides a basis for interpreting hazard rates in the continuous time model, where the dataset consists of monthly observations spaced at equal intervals $\Delta = 1$.
We use the full-solution maximum likelihood approach to estimate the model. We fixed the discount rate at $\rho = 0.05$ and the number of mileage states at $K = 90$. The value functions are obtained through value function iteration for each value of $\theta$ in an inner loop to within a tolerance of $\epsilon_V = 10^{-16}$ under the relative supremum norm. We maximized the likelihood function in an outer loop using the L-BFGS-B algorithm byrd95limited, zhu97algorithm with central finite difference derivatives with an adaptive step size proportional to $\epsilon_d^{1/3}$, with $\epsilon_d = 10^{-8}$. For robustness to local optima, we took the estimates to be the parameter values which achieved the highest likelihood over 20 random starting values.
Although this approach is straightforward, it is computationally intensive. For each iteration, it requires solving the fixed point problem for each trial value of $\theta$ and for small steps in the direction of each component of $\theta$. Alternative methods, such as the NFXP approach by rust87optimal, utilize analytical derivatives of the Bellman operator to compute analytical derivatives of the log-likelihood function. Combined with the BHHH algorithm bhhh-1974, which approximates the Hessian of the log-likelihood via the outer product of scores using the information matrix identity, this provides substantial computational savings in models with many parameters. Although the Bellman operator in continuous time is differentiable, this requires computing analytical derivatives of the matrix exponential with respect to individual components of the matrix argument. These methods are computationally expensive, involving truncation of infinite sums, evaluation of numerical integrals, or eigenvalue decompositions of high-dimensional matrices magnus-pijls-sentana-2021. On the other hand, Newton-Kantorovich-type methods could have particular advantages in the context of continuous time models, where the $Q$ matrix is typically very sparse, leading to sparse derivatives of the Bellman operator.
The estimated structural parameters and standard errors are reported in Table (ref). The first column of results corresponds to the model where we hold fixed $\lambda = 1$ (i.e., $\lambda_{\text{H}} = \lambda_{\text{L}} = 1$). In this model, the manager is assumed to make decisions on average once per month, corresponding to the timing of decisions in a discrete time model. The second column contains estimates for the model where we allow $\lambda$ to vary and estimate it (i.e., $\lambda = \lambda_{\text{H}} = \lambda_{\text{L}}$). The final column reports estimates for the heterogeneous version of the model where $\lambda_{\text{H}}$ may differ from $\lambda_{\text{L}}$.
In the variable $\lambda$ model, we can see that the estimated decision rates are quite different from 1. Therefore, this provides an interesting setting in which to compare the estimated costs and differences in interpretation. The variable $\lambda$ specification indicates a relatively low rate of monitoring, with $\hat\lambda = 0.032$ (vs. $\lambda = 1$), but a higher cost of mileage, $\hat\beta = -1.257$ (vs. $\hat\beta = -0.533$). When $\lambda$ is constrained to 1 (forcing monthly decision-making), the model compensates by estimating a lower mileage cost to match the observed replacement timing. A manager who checked frequently but with high mileage costs would replace the engine more often than observed in the data.
In the heterogeneous model, there appears to be a slight decrease in the estimated rate of monitoring in lower mileage states, with $\hat\lambda_{\text{L}} = 0.022$ for lower mileage states as compared to $\hat\lambda_{\text{H}} = 0.033$ in high mileage states. The estimated cost of mileage is $\hat\beta = -1.711$ and the cost of replacement is $\hat\mu = -9.643$.
To choose between these three nested specifications, we carry out likelihood ratio tests of the null hypotheses of homogeneity, $H_0: \lambda_{\text{H}}=\lambda_{\text{L}}$, and decision rates on average equal to monthly decisions in the discrete time model, $H_0: \lambda = 1$. We fail to reject the homogeneity restriction $\lambda_{\text{H}} = \lambda_{\text{L}}$, but strongly reject the specification with $\lambda = 1$. It appears to be important to let the rate of decisions vary as a parameter to be estimated, but perhaps they are constant across mileage states in this setting.
\@startsection{subsection}{2}{\z@} {-3.25ex\@plus -1ex \@minus -.2ex} {1.5ex \@plus .2ex} {\normalbaselines\normalfont}{Single Agent Renewal Model: Monte Carlo}
Inspired by the estimates above, we conducted a Monte Carlo experiment using the model with true parameters specified as follows: $(\lambda_{\text{L}}, \lambda_{\text{H}}, \gamma, \beta, \mu) = (0.05, 0.10, 0.5, -2.0, -9.0)$. We also report estimates of the cost ratio $\mu/\beta = 4.5$ which, as is common in discrete choice models, is more precisely estimated in most specifications than $\beta$ or $\mu$ individually.
In the Monte Carlo, we estimate the model under several different sampling regimes including full continuous-time data and discrete time data sampled at short and long intervals $\Delta = 1$ and $\Delta = 8$. Recall that in the real dataset, $\Delta = 1$ corresponds to a time period of one month. In the simulation, we can interpret $\Delta = 8$ as observing the manager's decision only once every 8 months. We simulate data over a fixed time interval $[0,T]$ with $T = 120$ months for each of $M$ markets, with $M$ varying from $200$ to $3,200$. Recall from Table (ref) that the maximum time horizon was $T = 125$, so our simulation time horizon is slightly shorter. Similarly, in the actual dataset we observed $M = 162$ buses. Our simulated small sample size is $M = 200$, and we increase that to $M = 800$ and then $M= 3200$ to evaluate the large sample properties of the estimator.
For each specification, we report the mean and standard deviation of the parameter estimates over 100 replications in (ref). With the smallest sample size, $M=200$, although the rate parameters $\lambda_{\text{L}}$, $\lambda_{\text{H}}$, and $\gamma$ are quite precisely estimated in all cases---even with a long time interval between discrete time observations---the cost parameters $\beta$ and $\mu$ are overestimated. However, even still, they are overestimated in a way such that the ratio $\mu / \beta$ is close to the true value. In large samples---as we increase the sample size to $M = 800$ and $M = 3200$---all are parameters are estimated quite precisely and with little bias. The loss of precision, measured by the standard deviations of the parameter estimates, is minimal when moving from continuous-time data to discrete-time data with $\Delta = 1$, but it becomes more noticeable at $\Delta = 8$.
\@startsection {section}{1}{\z@} {-3.5ex \@plus -1ex \@minus -.2ex} {2.3ex \@plus.2ex} {\normalbaselines\normalfont}{Conclusion}
In this paper, we developed new results on the theoretical and econometric properties of a generalized instance of the empirical framework introduced by \citet*{abbe-2016} for continuous time dynamic discrete choice games. We showed that the rate of move arrivals can be identified, even in models where it is heterogeneous, depending on the player identity or state, whereas previously it was assumed to be known. We established equilibrium existence with heterogeneous players and state-dependent move arrival rates, developed conditions for identification with discrete time data in the more general model, explored these results in the context of three canonical examples widely used in applied work, and examined the computational properties of the model as well as the finite- and large-sample properties of estimates through a series of small- and large-scale Monte Carlo experiments based on familiar models.