Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
73,798 characters · 12 sections · 59 citation commands
Causal Inference under Interference through Designed Markets
{\it Keywords: } School Choice, Econometrics of Auctions, Spillovers
\setstretch{1.7}
An individual-level intervention in an economic system rarely affects agents in isolation. Interactions among market participants lead to spillover effects, where the treatment of one individual affects the outcomes of others. Spillover effects make it challenging to estimate global treatment effects, such as the difference in expected outcomes when everyone is treated compared to when nobody is treated ($\bar \tau_{\text{GTE}}$). Existing approaches in the causal inference literature assume either partial interference, where there are no spillovers across clusters of agents baird2018optimal, hudgens2008toward, or that spillovers occur through an observed network where connections between agents are sparse aronow2017estimating, leung2020treatment. Except for parametric model-based approaches, there has been limited progress in estimating global treatment effects under complete interference, where the treatment of any individual may impact anyone else's outcome. We show that in markets where spillover effects are mediated by a specific class of centralized allocation mechanisms, even though there is complete interference, semi-parametric estimation of global effects is possible.
Settings where a centralized mechanism allocates scarce items are increasingly common in practice. In the U.S., versions of the deferred acceptance algorithm allocate students to schools abdulkadirouglu2003school, and medical school graduates to residency programs roth2003origins. Auctions allocate advertisements to search queries varian2014vcg and Treasury bonds to investors mcmillan2003market. Often, policymakers are interested in estimating how an intervention that affects the reported preferences of market participants will impact resulting allocations from the mechanism. For example, allende2019approximating provide information about school quality to families in a randomized experiment in Chile, where a centralized mechanism determines allocations to schools. One of their target estimands is $\bar \tau_{\text{GTE}}$, where the treatment is the information intervention and the outcome is the allocation of low-income families to high quality schools.
Estimators of the Average Treatment Effect fail to recover $\bar \tau_{\text{GTE}}$, even when the treatment is randomly assigned. By increasing the number of applicants to schools with limited capacity, the treatment affects admissions probabilities, which introduces spillovers and violates the Stable Unit Treatment Value Assumption (SUTVA) heckman1998general. To estimate $\bar \tau_{\text{GTE}}$, allende2019approximating use data from the experiment to estimate a parametric structural model of reported preferences over schools, and simulate the relevant counterfactuals using this model and the centralized mechanism.
In this paper, we take a new approach, which derives a sufficient-statistics representation of the GTE, and requires only data from a single market where the treatment is quasi-randomized. This is a step toward extending more “credible" approaches based on quasi-experimental variation angrist2009mostly to a richer set of counterfactuals beyond the average treatment effect (ATE). We begin with a potential outcomes model that allows for complete interference. We then make three major restrictions under which $\bar \tau_{\text{GTE}}$ is identified for any mechanism: we assume that SUTVA holds at the level of individual reports to the mechanism, outcomes can be computed from the mechanism, and treatments follow selection-on-observables. While these assumptions can be relaxed, doing so restricts the validity of the estimand to certain subgroups. For the unrestricted $\bar \tau_{\text{GTE}}$, our identification result suggests a plug-in approach for estimation: estimate the distribution of counterfactual submissions to the mechanism (bids) non-parametrically, and then run the mechanism on samples drawn from these distributions. Depending on the properties of the allocation mechanism, however, this approach may have an unacceptably large variance in finite samples.
For an estimator with better properties, we restrict attention to mechanisms that have a cutoff representation azevedo2016supply, agarwal2018demand. This class of mechanisms has an equilibrium which is defined by a finite vector of market-clearing cutoffs, and includes the uniform price auction, deferred acceptance, and top trading cycles. Even with a cutoff mechanism, the observed market may have multiple equilibria, and $\bar \tau_{\text{GTE}}$ is an average of interdependent terms. Under an asymptotic framework where a finite-sized market with $n$ participants converges to a continuum market with infinite participants azevedo2016supply, we show that the finite market $\bar \tau_{\text{GTE}}$ converges at a $1/\sqrt n$ rate to a continuum market counterfactual, $\tau^*_{\text{GTE}}$. The continuum market counterfactual has a simple representation in terms of a set of moment conditions defined on the distribution of submissions to the mechanism.
Our estimator solves an empirical version of the moment condition representation of the continuum market GTE. It relies on doubly-robust scores, and requires a careful adaptation of results on localization approaches for GMM models with missing data, specifically the work of kallus2019localized. In the first step, we use a propensity-score approach to estimate counterfactual market-clearing cutoffs. In the second step, a debiased estimate of counterfactuals is computed by running a re-weighted and perturbed version of the mechanism, where the perturbations are estimated using a simple set of machine learning regressions on the first-stage estimates. Data-splitting is used to control bias, allowing for weak conditions on the convergence rates of the machine learning estimators.
Using techniques from the theory of empirical processes, we show that the estimator is asymptotically normal, and that inference valid for the continuum market counterfactual is conservative for the finite-market estimand. This means that the estimator is robust to a variety of specifications for how bids are affected by the treatment -- as long as certain statistics of the counterfactual bid distributions are sufficiently smooth and the machine learning estimators meet regularity conditions, we can perform inference on $\bar \tau_{\text{GTE}}$. Furthermore, the variance of the estimator meets the semi-parametric efficiency bound for the continuum market counterfactual, which suggests that our inference approach has good power compared to alternative approaches.
Another advantage of our semi-parametric approach is that we allow for unrestricted heterogeneity in treatment response. When treatment effects are heterogeneous, a policymaker can improve welfare by assigning treatment to a subset of individuals, depending on their pre-treatment covariates. There is a large literature on policy learning under SUTVA, but the problem is much more complex when there are spillover effects, as discussed in the network setting by vivianorestud. In this paper, we provide the first asymptotic regret results for policy learning with market spillovers, employing a two-step, doubly-robust approach for empirical welfare maximization. This yields an asymptotic regret bound in the finite market that is of the same order as the lower bounds in the literature on policy learning without spillover effects athey2021policy. Constraining spillovers to occur through the mechanism and knowing the structure of the mechanism is crucial for this result. A major step in the proof, which is the most challenging technical result of the paper, is demonstrating uniform convergence of estimated market-clearing cutoffs to the continuum market-clearing cutoffs.
In simulations of a uniform price auction, we illustrate the robustness properties of our preferred estimator, in contrast to approaches based on parametric structural modeling. Finally, we apply our methods in a real-world setting using data from Chile, where children are allocated to public schools via a version of deferred acceptance. We compile a dataset from the Ministry of Education that replicates many of the features of the data in allende2019approximating, except that the treatment is self-reported receipt of government-provided information on school quality, rather than an explicitly randomized intervention. We estimate $\bar \tau_{\text{GTE}}$, where the outcome measures the allocation of low-income families to good-quality schools. We find that if spillover effects are ignored, then the estimate of the impact of the treatment is significant, raising access of low-income families to good schools by nearly 1.5 percentage points. However, an estimate of the true impact of the intervention that takes into account the impact on the equilibrium of the school market is much smaller at 0.5 percentage points. A rule approximating the optimal targeting rule in equilibrium raises access of low-income families to good schools by 1.8 percentage points, substantially outperforming a uniform rule that allocates the intervention to all families.
There is a body of existing work that estimates different types of causal effects in designed markets. abdulkadirouglu2017research estimate causal effects of allocations on future outcomes, such as test scores or income, using randomness in the matching mechanism for identification. abdulkadirouglu2022breaking, chen2021nonparametric, and bertanha2023causal extend this work to settings where individual scores are non-random but the cutoff structure of the mechanism allows an RDD analysis. bertanha2023causal also considers partial identification of preferences from strategic reports when mechanisms are not strategy proof. In contrast to this body of work, our paper focuses on an earlier step in the causal chain of events, which is the effect of a pre-allocation intervention on resulting allocations.
athey2007nonparametric survey non-parametric identification and estimation methods for primitives in a wide range of auction models. We focus on the estimation of specific counterfactuals rather than model primitives. The disadvantage of our approach is that estimating primitives is useful for estimating a wider range of counterfactuals, and can sometimes be of independent interest. The advantage is efficiency and robustness, in that we can obtain precise estimates without imposing strong (e.g. parametric or distributional) assumptions, at least with strategy-proof mechanisms.
Sufficient statistics approaches are popular in a variety of applied economics fields, including public economics chetty2009sufficient and macroeconomics mckay2023can. The approach in this paper is unique in that most of the key assumptions that lead to the sufficient statistics representation are known properties of the market mechanism, rather than parametric or distributional assumptions imposed by the researcher on the data generating process. In addition, we provide theoretical guarantees on inference and robustness of our method, which are not always available in the related literature. There is a small literature in causal inference that considers settings where interactions occur through a known algorithm or statistic. miles2019causal studies a model where spillovers occur only through the proportion treated. bright2022reducing characterize the bias of an RCT in a parametric model of a matching market, where a linear program computes the matching. They propose a simulation-based estimator of the GTE that requires estimating their model using maximum likelihood estimation. Our paper studies markets with a different class of matching mechanisms that are truthful and have a cutoff structure; in this class of mechanisms, we estimate causal effects without imposing a parametric model of behavior.
munro2023market also constrain spillovers to occur through a set of market statistics. Without the presence of a centralized mechanism, the model primitives are the distribution of demand and supply functions, rather than counterfactual distributions of bids. This means the authors are limited to counterfactuals that are local to the current equilibrium, and require more complex experimental designs with price randomization for identification. In contrast, the current paper is more directly related to the literature on structural modeling in that a) we use data with standard treatment variation and b) we identify global treatment effects that extrapolate from the observed market. Furthermore, the methods for handling nuisance function estimation and uniform convergence are novel compared to the approach in munro2023market.
To analyze the properties of the estimators in the paper, we use an asymptotic framework where the allocation mechanism operates on a continuum of agents rather than a discrete number of agents. Large-sample approximations have been used to characterize the bias and variance of A/B testing-based approaches in online marketplaces in johari2022experimental, bright2022reducing and liao2023statistical.
In the market observed by the researcher, $n$ participants are allocated using a centralized mechanism to some subset of $J$ items, which each have limited capacity. For each market participant, the researcher observes pre-treatment covariates $X_i \in \mathcal X$, a binary treatment $W_i \in \{0,1 \}$ and a submission to the mechanism $B_i = B_i(W_i)$, which may be affected by the treatment. Individuals with $(X_i, B_i(1), B_i(0), W_i)$ are drawn i.i.d. from some distribution $F$. Individual allocations $D_i = D_i(\bm W) \in \{0, 1\}^J$ and outcomes $Y_i = Y_i(\bm W) \in \mathcal Y$ depend on other individuals' treatments and actions through a centralized mechanism\footnote{Although our primary examples in the paper have binary allocations, the analysis in the paper extends directly to allocations that are integers or real numbers, as long as they are bounded.}.
In this paper, we will estimate and maximize the value of counterfactual treatment rules. A candidate treatment rule is a function $\pi: \mathcal X \to [0, 1]$, where $\pi \in \Pi$. Treatment allocation under the counterfactual rule is $W_i \sim \mbox{Bernoulli}(\pi (X_i))$. The finite-market value of a counterfactual treatment rule is defined as the expected outcomes for a market of $n$ participants, \[ \bar V_n(\pi) = \frac{1}{n} \sum \limits_{i=1}^n \mathbb E_{\pi} \left [ Y_i(\bm W) \right ], \] where $E_{\pi}[\cdot]$ is the expectation with respect to random treatment allocation, conditional on the realized potential outcomes and covariates. Although the theory in Section 3 allows us to estimate the difference in average value between any two candidate policies, we pay particular attention to the Global Treatment Effect $(\bar \tau_{\text{GTE}})$, which is defined as the difference in welfare when everyone is treated compared to when no one is treated: $ \bar \tau_{\text{GTE}} = \frac{1}{n} \sum \limits_{i=1}^n Y_i(\bm 1_n) - Y_i(\bm 0_n),$ where $\bm 1_n$ and $\bm 0_n$ are $n$-length vectors of 0s and 1s. We start by providing a series of assumptions under which we can identify the value of counterfactual treatment rules using the joint distribution of $(B_i, X_i, W_i)$. Let $\bm B(\bm w)$ be the $n$-length vector of submissions to the mechanism under treatment vector $\bm w$.
In the first part of Assumption (ref), we assume that the mechanism is known, so that an individual's allocation $D_i(\bm w) \in \mathbb R^J$ at a given treatment vector $\bm w \in \{0, 1\}^n$ can be computed given the $n$-length submissions to the mechanism $\bm B(\bm w)$. This assumption holds for any market that is cleared by an auction or matching mechanism. We also assume outcomes can be computed from $\bm B(\bm w)$, which is the case for bidder surplus in auctions and the measure of inequality in school allocations studied in Section (ref). This framework extends to general outcomes like test scores or income, under the assumption that $Y_i(\bm W) = \sum \limits_{j=1}^J M_{j}(X_i) D_{ij}(\bm W) + \epsilon_i$ is a weighted sum of item-specific effects, as in the literature on school value-added AbdulkadirogluPathakWalters2025, if $M_j(\cdot)$ is known. Weakening this assumption to handle outcomes that are unknown functions of allocations is also possible for some mechanisms by combining the approach in this paper with the lottery-based identification strategy of abdulkadirouglu2017research. However, this comes at the cost of identifying a restricted version of $\bar\tau_{\text{GTE}}$ that is valid only for certain subgroups; see Appendix (ref).
The last part of Assumption (ref) identifies the marginal distribution of $B_i(1)$ and $B_i(0)$ by assuming that the treatment is randomly assigned, conditional on covariates.\footnote{It is possible to use an IV-type assumption as an identifying condition instead at the cost of only identifying a restricted version of $\bar \tau_{\text{GTE}}$, see Appendix (ref).} Under this set of assumptions, $\bar \tau_{\text{GTE}}$ is identified, and is a known functional of the treatment rule $\pi(\cdot)$, the distributions of $(B_i(1), X_i)$ and $(B_i(0), X_i)$, and the market size. A natural next step is plug-in estimation: first, estimate counterfactual distribution of bids non-parametrically, and then run the mechanism on samples from these distributions. Depending on the properties of the mechanism, the estimand may depend on features of the distribution that are infeasible to estimate non-parametrically in finite samples, especially when $X_i$ or $B_i$ are high-dimensional.\footnote{In the school choice setting, the number of possible submissions to the mechanism is exponential in the number of schools.}
Rather than pursuing plug-in estimation, which may converge extremely slowly or not at all, we instead specify a general class of economic mechanisms where a $\sqrt n$ convergent and computationally efficient estimator for $\bar \tau_{\text{GTE}}$ is available. This class, formalized in Assumption (ref), is made up of mechanisms for which an individual's allocation depends only on their own submission to the mechanism and a set of market-clearing cutoffs. A variety of commonly-used mechanisms have a cutoff structure, including the uniform price auction, deferred acceptance azevedo2016supply, and top trading cycles leshno2021cutoff. In munro2023market, restricting spillovers to occur through market prices is useful for identification of local treatment effects in general equilibrium. Although our identification strategy is entirely different, limiting the complexity of interactions that occur through the mechanism is still necessary to show that estimators of $\bar \tau_{\text{GTE}}$ converge at the parametric rate.
For prices, we use notation that makes explicit the dependence of market-clearing cutoffs on the vector of treatments. Although it is not explicit in the notation, prices also depend on bids and characteristics of the market. The cutoffs are computed by the mechanism and need not be unique. Formally, there exists an algorithm, represented by a function $m: \mathcal B^n \times \Delta^{n-1} \times [0, 1]^J$ that maps the $n$-length vector of bids $\bm B(\bm w)$, an $n$-length vector of weights for each bid $\bm \gamma$, and capacities for each item to a market-clearing cutoff, so
and we can write $P_n(\bm w) = m( \bm B(\bm w), \bm X, \frac{1}{n} \cdot \bm 1_n, s^*)$, where $\Delta^k$ is the $k$-dimensional simplex. This concept of market-clearing cutoffs with possibly heterogeneous weights is useful for estimating counterfactuals in the next section. We next introduce two examples of mechanisms that are regularly used in practice and have such a cutoff structure.
Under Assumption (ref), then:
\[ \bar \tau_{\text{GTE}} = \frac{1}{n} \sum \limits_{i=1}^n y(B_i(1), X_i, P_n(\bm 1_n)) - y(B_i(0), X_i, P_n(\bm 0_n)). \] In a finite market, the mechanism allocates a fraction of the empirical distribution of market participants to each item. The equilibrium may not be unique, and furthermore, counterfactuals are defined in terms of averages of dependent terms, since $P_n(\bm W)$ depends on all market participants. We next introduce the continuum market, which is a useful approximation of the finite market that allocates an equivalent fraction of the population distribution of market participants to each item azevedo2016supply. Continuum market counterfactuals are defined in Definition (ref) as a simple set of moment conditions and have a unique equilibrium under a straightforward set of conditions in Assumption (ref).
To conclude this section, we show that not only does the continuum market provide a sufficient-statistics representation of the counterfactual, it is also a good approximation asymptotically of the finite market. Our notion of convergence follows the related economic theory literature, as in azevedo2016supply, and takes both total supply and $n \to \infty$ but keeps $J$ and fractional supply $s^*$ fixed. This asymptotic approximation relies on a representation of a mechanism as a functional of the empirical distribution of market participants. Then, it replaces the empirical distribution of market participants with the smooth population distribution. It is a good approximation in finite samples as long as supply for each product is not too small.
We impose a set of regularity conditions that ensure that the finite and continuum markets are sufficiently well-behaved. In Assumption (ref), the weak continuity assumption and metric entropy condition allow for individual-level allocation functions that have some discontinuity in market-clearing cutoffs. However, at the population level, expected allocations and outcomes must be smooth.
In Assumption (ref), we assume that the market-clearing cutoffs in the population are unique and well-separated. Under conditions on the smoothness of the distribution of values, Assumptions (ref) - (ref) are satisfied by the uniform price auction in Example (ref), when bidder surplus is the outcome of interest, as shown in Appendix (ref). This result can also be extended to Example (ref) under regularity conditions on the distribution of lottery numbers in deferred acceptance.
Under these assumptions, our first result strengthens the convergence result in azevedo2016supply by providing a rate at which counterfactuals in the finite market converge to those in the continuum market. As the market size grows large, the value of a treatment rule in equilibrium converges from an average of dependent terms to a set of moment conditions defined on the population distribution.
We prove Theorem (ref) in Appendix (ref) using techniques from empirical process theory vaart1997weak. Using related techniques, munro2023treatment shows convergence of a local equilibrium effect to a large-market approximation, but does not provide a rate. By providing a rate, Theorem (ref) provides a foundation for inferential guarantees for the estimator of $\bar V_n(\pi)$ introduced in the next section.
The previous section established a moment-based approximation for cutoff mechanisms, eliminating the dependence, under general mechanisms, of the estimand on complex features of the distribution of bids. Using this representation, we next provide a doubly-robust estimator that is $1/\sqrt n$-consistent for $\bar \tau_{\text{GTE}}$. Unlike existing semi-parametric methods that rely on complex experimental designs munro2023market, bajari2023, the estimator relies only on data from a standard RCT. Algorithmically, our estimator runs a perturbed and re-weighted version of the allocation mechanism on the observed data, where the weights and perturbations are estimated using flexible machine learning methods and three-way data splitting is used to control bias. This estimator is closely related to the more general theory in kallus2019localized for quantile-like treatment effects, but aspects of its design and analysis are unique to the problem studied in this paper.
Combining the moment representation of $V^*(\pi)$ and the overlap and unconfoundedness assumptions of Assumption (ref), we can identify $V^*(\pi)$ using $J+1$ moment conditions and doubly-robust scores:
where doubly-robust scores combine the propensity score $e(x) = P(W_i = 1| X_i = x)$ and conditional mean functions $ \mu^d_w(x, p) = \mathbb E[d(B_i(w), X_i, p) | X_i = x]$ and $ \mu^y_w(x, p) = \mathbb E[y(B_i(w), X_i, p) | X_i = x]$ for $w \in \{0, 1\}$:
Equation (ref) is not the only set of moment conditions that identify $V^*(\pi)$ under unconfoundedness and overlap. For example, it is possible to identify and estimate $V^*(\pi)$ using the propensity score only. We prefer the doubly-robust approach since it requires much weaker assumptions on propensity scores for results on inference and semi-parametric efficiency. For a more detailed discussion of the benefits and drawbacks of the propensity score approach, we defer to the large related literature bang2005doubly, graham2012inverse. Another alternative, which is popular in the applied economics literature and discussed in more detail in Section (ref), is to use a parametric structural model of bidding behavior for identification and estimation. Our approach avoids specifying a parametric model of bidding behavior.
The simplest doubly-robust estimator would solve for an empirical version of (ref), as in chernozhukov2018double. However, this requires inverting the estimated conditional mean function, since it is a function of $p$, which implies estimating the entire bid distribution conditional on covariates. When the bid or covariate dimension is high, a flexible estimator of this conditional distribution will converge too slowly for the theory in chernozhukov2018double. Instead, we adapt the localization approach of kallus2019localized, which solves an empirical version of (ref) that fixes the cutoff component of the conditional mean functions at a first-step estimator of counterfactual market-clearing cutoffs. An application of this approach that uses the centralized mechanism $m(\cdot)$ to find a solution to the empirical moment condition is in Definition (ref).
Data are split three ways. The first split estimates a pilot value for the counterfactual cutoffs, the second split estimates nuisance functions at that pilot cutoff, and the third evaluates scores. In settings like school choice where $J$ can be large, root-finding algorithms may be computationally prohibitive. The localized approach uses the mechanism $m(\cdot)$ to find the market-clearing cutoffs, where demand is re-weighted to account for selection-on-observables, and supply is perturbed to adjust for bias in the first-step estimates.
A structural approach usually imposes a parametric assumption on the distribution of bids conditional on covariates; once the parameters of that model are estimated, counterfactuals can be simulated directly from the model. The advantage of the approach in Definition (ref) is that it relies only on weak assumptions on the estimators of the propensity score and a set of conditional mean functions. Under Assumption (ref), the doubly-robust estimator is asymptotically normal and semi-parametrically efficient.
Assumption (ref) requires that the pairwise product of the convergence rates of the initial estimator of the counterfactual cutoffs, the propensity score, and the conditional mean functions are $o(n^{-1/2})$. This means that for a fixed $p$, the estimator for expected outcomes and allocations conditional on $X_i$ can have a slow mean-square convergence rate. The uniform guarantee on the performance of the estimators over $\pi \in \Pi$ can be dropped for the point-wise results on the value function in this section, but is required for the regret guarantee in the next section. The main result of this section is that the algorithm described leads to an asymptotically normal estimator:
With known nuisance functions, standard techniques for method-of-moments estimators can be used to prove Theorem (ref) for a propensity score-based estimator. With an unknown propensity score and a doubly-robust estimator, the challenge is to show that even when estimated nuisance functions depend on market-clearing cutoffs, their estimation error does not have a first order impact on the error of the estimator.\footnote{Under weaker entropy conditions than in Assumption (ref), the main result in kallus2019localized can be used to prove Theorem (ref). However, the stronger conditions that we impose, which are met by economic mechanisms used in practice, lead to a more concise proof of Theorem (ref), and are useful for the regret results in Section (ref).}
Corollary (ref) follows directly from Theorem (ref). Due to the market-clearing cutoffs, the asymptotic variance of $\hat \tau_{\text{GTE}}$ depends on the variance of a linear combination of treatment effects on outcomes and treatment effects on allocations. The first component is the standard sampling variation in direct treatment effects, and the second is due to the variation in the equilibrium that is reached in the allocation mechanism. Furthermore, the variance in Corollary (ref) meets the semi-parametric efficiency bound for $\tau^*_{\text{GTE}}$.
If one is willing to impose a parametric assumption on the bid distribution, then a structural estimator of $\tau^*_{\text{GTE}}$ will be efficient. However, in the absence of a parametric assumption on how the treatment impacts bidding behavior, then the proposed estimator is semi-parametrically efficient. The proof of this theorem is in Appendix (ref). The proof uses the methodology presented in bickel1993efficient and newey1990semiparametric, and is closely related to the bound for quantile treatment effects in firpo2007efficient.
By computing a plug-in estimator of $\sigma^2$, we can perform asymptotically valid inference on the continuum market counterfactual $\tau^*_{\text{GTE}}$. Consistency of a plug-in estimator for $\sigma^2$ follows from the existing assumptions, as shown in Theorem 4 of kallus2019localized. Appendix (ref) uses Monte Carlo simulations to illustrate the finite-sample properties of confidence intervals based on the normal approximation of Corollary (ref). Theorems (ref) and (ref) focus on the continuum market counterfactual $\tau^*_{\text{GTE}}$. Although it is a convenient approximation, in many settings the true target of interest is the finite-market counterfactual $\bar \tau_{\text{GTE}}$. Combining Theorem (ref) and Theorem (ref), we have that $\sqrt n (\hat \tau_{\text{GTE}} - \bar \tau_{\text{GTE}}) \rightarrow_D N(0, \bar \sigma^2), $ where $\bar \sigma^2 = \mathbb E[ (\Gamma^{*q}_{1i}(p^*_1) - \Gamma^{*q}_{0i} (p^*_0) - q_1(B_i(1), X_i, p^*_1) + q_0(B_i(0), X_i, p^*_0) )^2 ] $. Proposition (ref) shows that inference that is valid for the continuum market estimand is conservative for the finite market estimand, which is the primary counterfactual of interest for a policymaker or market designer.
The model in Section (ref) allows for heterogeneity in the effect of the treatment on individual welfare. So far, however, the counterfactuals considered treat all market participants the same. In some settings where there is significant heterogeneity in treatment response, a market designer or policymaker may consider treatment rules that target some subset of market participants. In this section, we consider the problem of choosing $\pi \in \Pi$ to maximize finite market or continuum market expected outcomes. Because of interactions through the centralized mechanism, the benefit of treating a group of individuals depends on their direct response to the treatment as well as indirect effects on others; the magnitude of both can vary depending on the treatment saturation in the sample. In this paper, because the indirect effect is mediated by a known algorithm, learning optimal treatment rules is possible.
We start by characterizing the optimal unrestricted treatment rule in the continuum market; although this leads to a useful description of the structure of the globally optimal rule, designing an estimator with good theoretical guarantees requires additional assumptions. We then consider the problem of estimating a treatment rule that is a member of a restricted class of rules, and maximizes outcomes in the finite market. We restrict $\Pi$ to be a VC class and show that maximizing the estimated value function within this class using the algorithm in Section (ref) has regret that decays at a $1/\sqrt n$ rate. This is a notable result -- when interactions are mediated by a cutoff mechanism, it is possible to learn the optimal policy at an asymptotic rate that matches the lower bound for policy learning without spillover effects athey2021policy.
Theorem (ref) provides a score condition that any optimal rule must satisfy when $\Pi$ is unconstrained.
The score $\rho(x, \pi)$ consists of two components. The first captures the direct impact of treating participants with $X_i = x$, while holding market-clearing cutoffs fixed. The second accounts for the general equilibrium effect through capacity constraints, where $v^*_{\pi}$, represents the marginal social cost of the resulting demand shift on the outcomes of other participants. If the sum of these two effects is positive then the treatment probability for the group is positive. This is in contrast to the globally optimal rule under SUTVA, where only the sign of the conditional average direct effect of the treatment on outcomes matters. While this result is useful for understanding the structure of the optimal rule, the ultimate goal in this section is to characterize the regret of an estimator for the optimal treatment rule. Unfortunately, obtaining even consistency is challenging for the globally optimal rule; a plug-in estimator may not meet the condition of Theorem (ref), since $\hat \rho(x, e)$ estimated at the treatment rule observed in the data may be very different from $\rho(x, \tilde \pi)$, where $\tilde \pi(x) = \mathbbm{1}( \hat \rho(x, e) > 0)$. In the next section, we constrain $\Pi$ to be a VC class, which allows for an empirical welfare maximization approach that has asymptotic regret guarantees even in the finite market. Furthermore, this constraint is often useful in practice where “simple" treatment rules, such as linear threshold rules, are desirable.
As in kitagawa2018should, we now assume that $\Pi$ is a VC class of functions with dimension $v$. The estimator of the optimal value function maximizes the doubly-robust estimator of the value function from Section (ref) over $\Pi$, specifically $ \hat \pi \in \arg \max \limits_{\pi \in \Pi} \hat V_n(\pi). $
The main contribution of this section is formalizing how well the estimated rule performs compared to the oracle rule that maximizes the unobserved finite-market value $\bar V_n(\pi)$ directly. A key step in this result is to show that both $\bar V_n(\pi)$ and $\hat V_n(\pi)$ converge uniformly in $\pi \in \Pi$ as $n$ grows large to the continuum market value $V^*(\pi)$. For this uniform convergence, we require Assumption (ref), which is an additional assumption on the nuisance functions.
Although we allow the estimated conditional mean functions to be complex functions of $X_i$, they must be relatively simple functions of $p$. Since we already impose a metric entropy condition on individual-level outcome functions in $p$, in some cases, such as for the $K$-nearest-neighbors estimator used in Section (ref), this is automatically satisfied by Assumption (ref). For more general machine learning estimators, verifying this type of condition may require additional effort. We can now prove Theorem (ref).
Characterizing the maximizer of the finite-market value of a treatment rule directly is challenging, since it is a quantity that depends on possibly non-unique market-clearing cutoffs and non-smooth allocation functions. By linking both the finite-market value and estimated market-value to the continuum market value instead, where the equilibrium is unique and aggregate responses are smooth, then we manage to obtain asymptotic regret results for the finite-sized market. The constants in the asymptotic regret bound depend on the VC class dimension $v$, the number of items $J$, as well as the parameters $h_d$ and $h_y$ in the covering number bounds for the allocation and outcome functions. The dependence on $n$ implies that the estimated maximizer converges quickly to the oracle maximizer of either the finite or continuum market value. This rate matches the lower bound for policy learning with SUTVA and the upper bounds for regret with network spillovers in vivianorestud. This strong result is possible in the centralized market setting because all interactions occur through a finite-length vector of market-clearing cutoffs. A key step in the proof is showing $1/\sqrt n$- uniform convergence of the estimated market-clearing cutoffs to the continuum market-clearing cutoffs under weak assumptions on the convergence of nuisance functions.
In this section, we illustrate the robustness properties of doubly-robust estimators compared to structural modeling approaches using a simulation of a uniform price auction where bidders' values are generated from different distributions.
We simulate data generated from a uniform price auction and compare the LDML estimator of $\bar \tau_{\text{GTE}}$ to alternative approaches. In the simulation, treatment affects bids to the auction. There is a $20$-dimensional set of covariates that is correlated with the bids and affects the probability of selecting the treatment. The auction has a fractional capacity of $0.5$ so the top 50% of bidders in the auction receive a single unit of the good. The treatment affects outcomes through a shift in the distribution of bids submitted to the auction and through a shift in the equilibrium market-clearing price. The outcome of interest is the observed average surplus for bidders in the auction, assuming that the bids submitted to the auction are equal to the values for the bidders. The data-generating process is explicitly described in Appendix (ref). For each bidder, we observe the bid $B_i$, the treatment $W_i$, and pre-treatment covariates $X_i$. We compute RMSE and bias for a variety of estimators when the target estimand is $\bar \tau_{\text{GTE}}$ by repeatedly sampling a finite-sized market of size $n=100$, $n=1000$ and $n=10,000$. These estimators take as input $(Y_i, B_i, W_i, X_i)_{i=1}^n$.
The estimators are as follows:
With only 100 datapoints, the noise in the estimation for methods that rely on estimating the distribution of bids directly is high. As the number of datapoints increases, the model-based estimator, which makes the correct parametric assumption on the bid distribution, converges the fastest. The bias-corrected structural model also performs well, although has increased variance since the bias correction adds noise when the model is correct. The LDML estimator does not make any parametric assumptions and instead uses flexible machine learning estimators for nuisance parameter estimation. It has an asymptotic distribution that does not depend on the estimation errors of the nuisance functions. The ATE estimator, which ignores the equilibrium effect of the treatment, has a large bias even as the sample size increases.
In the second set of simulations, we generate bids from a truncated normal distribution rather than a lognormal distribution. Otherwise, the data-generating process is the same. We compute the set of estimators, where we continue to use a lognormal based approach for the structural modeling estimators.
This time, the structural modeling approach performs poorly. The parametric assumption is incorrect, and as a result the outcome model is asymptotically biased. The SMDR estimator uses the propensity score to successfully remove the bias from the structural model. The LDML estimator does not make any parametric assumptions on the bid distribution and continues to perform very well here.
If a parametric model is correctly specified, then a maximum-likelihood estimator of that model is asymptotically linear and efficient. In addition, once the primitives of the model are specified and estimated, a variety of counterfactuals can often be evaluated, including those that are more complex than the estimand considered in this paper. The downside of this approach is if the model is not correctly specified, then the estimator of $\tau^*_{\text{GTE}}$ will be asymptotically biased. Unfortunately, it can be challenging to specify a parametric model that captures the complexity and heterogeneity of individual choice behavior, especially in settings where possible submissions to the mechanism are high-dimensional. The localized doubly-robust estimator performs well, without requiring correct specification of a parametric model of submissions to the mechanism.
In 2015, the Chilean government passed the Inclusion Law, which eliminated school-specific admissions criteria in favor of a centralized admission system based on deferred acceptance correa2019school. It was intended to reduce socioeconomic segregation in the Chilean school system by removing discriminatory admissions criteria and reserving some seats at good schools for low-income families. Despite these changes, low-income families attend good-quality schools at a much lower rate than high-income families.
One reason for this remaining gap is that some families may lack information about school quality or the returns to schooling. allende2019approximating explore this hypothesis using an RCT that randomized information on nearby school quality. They found that the intervention increases applications of low-income families to high-quality schools. Using a parametric model, they find that the effect on allocations in equilibrium is substantially less, due to capacity constraints.
We estimate and perform inference on the effect of information on income inequality by constructing a similar observational dataset on Chilean students. We also find that information affects choices positively, and that capacity constraints reduce the effect of the intervention on allocations significantly. We combine two datasets from the Ministry of Education for 2018 - 2020. For the admissions system, we use publicly available data on the centralized admissions process (SAE) for 2020 for those applying to the 9th grade in Chile. This data includes the rankings each student submits to the algorithm, their priority, location, and actual assignment. We link this to a demographic survey collected as part of the SIMCE\footnote{Sistema de Medición de la Calidad de la Educación} standardized test system in Chile. For school quality for the 9th grade admissions process, we use the average student math and reading score for the school in 2018 among 10th graders. Students apply to a subset of approximately 2,500 schools nationwide.
The treatment we analyze is a proxy for the receipt of information on government school quality. $W_i = 1$ if a parent responds “Yes" to the following question: \\ {\em Do you know the following information about your child's school? Performance category of this school}.\footnote{The survey language (in Spanish) is: ¿Conoce usted la siguiente información del colegio de su hijo(a)? Categoría de desempeño de este colegio. It is the third question in the thirtieth section of the parent survey in the SIMCE dataset.} Of the sample of 114,749 applicants to 9th grade, 53% have $W_i = 1$. The observed pre-treatment covariates are location, household size, mother and father education level, whether or not the mother and father are indigenous and the income of the family. Missing covariates are imputed using a k-nearest neighbors approach. Table (ref) in Appendix (ref) includes the mean and standard deviation for each of the variables.
We first check that the treatment impacts the rankings that low-income families submit to the allocation mechanism before we examine the effect on allocations. Submitted rankings are not subject to spillover effects through the allocation mechanism, since deferred acceptance is strategy-proof. So, we use DR-ATE to estimate the average treatment effect on two outcomes for low-income families in Table (ref). The first outcome is an indicator that is 1 if the family ranks a top 50% school first, and the second outcome is the length of the application list that a family submits. Note that the length of the submitted rankings is unrestricted in the Chilean mechanism. The estimated treatment effect on ranking a high-quality school is 2.3%.\footnote{In the market, 36% of low income families with $W_i = 0$ rank a top-50% school first.} The effect on list length is positive, but small. Thus, there is evidence that the information intervention encourages low-income families to apply to better-quality schools.
Because of capacity constraints, not all families that rank a high-quality school first are admitted to that school. Estimating treatment effects on allocations is more challenging due to spillovers that occur through the allocation mechanism. Table (ref) shows an estimate of treatment effects, when the outcome is whether a low income family is accepted to an above-average school in Chile. We see that the DR-ATE estimator, which corrects for selection, but not equilibrium effects, estimates a 1.3 percentage point increase in the allocation of low-income families to good quality schools. The LDML-GTE estimate is 0.5 percentage points, which is much lower. Figure (ref) provides a breakdown of the bias of the DR-ATE estimator. At the observed equilibrium, the probability of admission to a good-quality school is higher than at the 100% treated equilibrium and lower than that of the 0% treated equilibrium. Estimating $\bar \tau_{\text{GTE}}$ accurately requires estimating the access of treated families at the all-treated equilibrium, and control families at the all-control equilibrium.
We briefly discuss a possible source of bias in the LDML-GTE estimate. There are two possible sources of spillovers from an information treatment; the first is through the mechanism due to capacity constraints, and the second is network-related spillovers. The estimates in Table (ref) only account for the first type of spillover. Even if a family does not report receiving school quality information, they may make choices that are correlated with their treated neighbors' choices. If the network spillovers are positive, so that increasing the number of treated neighbors always increases the probability that a family raises the rank of a high-quality school, then the effect estimate in Table (ref) is a lower bound on the Global Treatment Effect under both network and congestion effects. If network spillovers may be positive or negative, then further work is needed to account for both types of spillovers.
By using a potential outcomes framework to analyze counterfactuals in this setting, heterogeneity in the effect of the treatment on bids is not restricted. There may be heterogeneity in whether or not individuals respond positively to the information, as well as heterogeneity in how these changes affect congestion in the centralized mechanism. As discussed in Section (ref) we can choose and evaluate treatment rules that treat only a subset of the sample defined by pre-treatment covariates.
Figure (ref) estimates the outcomes for a variety of treatment rules. All-Control assigns nobody to treatment and All-Treated assigns everybody to treatment. The Observed rule is the treatment pattern observed in the data. The targeting rule approximates a version of the globally optimal rule in Section (ref) through plug-in estimation and the value of the rule is estimated on a hold-out sample of the data.
The gain of the targeting rule over a rule that treats everyone is large, at 1.27% with an estimated standard error computed using the bootstrap of 0.46%. It also significantly outperforms a simple rule that assigns treatment only to low income families. This indicates that there is substantial heterogeneity in treatment response in the data.
It is not clear that in practice it would be desirable or fair to target the basic information on school quality considered in this specific example. However, the presence of significant heterogeneity in treatment response suggests that targeted policies may be of interest in school choice settings.
Without some structure, estimating causal effects with general spillovers is infeasible. Under a fully specified and point-identified parametric model of individuals interacting in a market, any counterfactual can be simulated, but the model must be specified correctly. In this paper, we instead use the structure implied by a centralized allocation mechanism, but remain non-parametric about individual choices, which can be difficult to specify correctly. Using a continuum market approximation of the finite market, we show that global counterfactuals in finite markets are well-approximated by a set of moment conditions. This leads to a computationally simple and doubly-robust estimator for the value of counterfactual policies. With data from the school market in Chile, we show that correcting for congestion effects substantially reduces the estimated effect of an information intervention on inequality in school allocations.
There are a variety of counterfactuals of interest that go beyond the estimands considered in this paper. These include settings with supply side responses and mechanisms with strategic behavior, where individuals make choices conditional on their expectations of the market equilibrium. For these problems, exploring whether it is possible to derive robust estimators that combine general causal models with economic structure imposed by design will be an interesting avenue for future work.
The data that support the findings of this study are available from the Chilean Ministry of Education. Restrictions apply to the availability of these data, which were used with permission for this study.
\singlespacing