Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
58,788 characters · 22 sections · 72 citation commands
Interpreting TSLS Estimators in Information Provision Experiments
There has been a surge in research that uses the random provision of information to estimate the causal effects of beliefs on actions.\footnote{In macroeconomics, researchers have studied the effects of beliefs about inflation, GDP growth, and other macroeconomic indicators on firm and household decision making coibion2020inflation, coibion2021effect, coibion2022monetary, coibion2023forward, kumar2023effect; in labor economics, researchers have studied the effects of beliefs about others' wages on one's own efforts and job search decisions cullen2022much, jager2022worker, and how beliefs about labor market tightness affect support for unions pezold_labor_2023; at the intersection of labor and public finance, researchers have studied the effect of beliefs about future government benefits on human capital investments deshpande2023lack; yet others have studied the effects of beliefs about discrimination on policy preferences haaland2023racial, settele_how_2022.} These information provision experiments provide a basis for testing the assumptions of economic models, differentiating across theoretical mechanisms, and informing economic policy haaland2023designing. Given the importance of these goals, it is essential that the causal parameters of interest be carefully defined and accurately estimated.
However, in practice, the causal parameters of interest are often informally defined, or are developed in stylized models that impose strong conditions on agents' beliefs, learning, and actions. In such models, it is unclear which of the implied restrictions drive the conclusions of an information provision experiment. Relatedly, it is ambiguous which of the many estimation strategies from the literature recover interpretable causal parameters. This paper addresses both of these concerns.
An information provision experiment generally proceeds as follows. First, the experiment elicits features (e.g., expectations) of agents' prior beliefs over a set of action-relevant states. Next, the experiment randomly assigns agents to different groups---either to a control group receiving no information, or to one of potentially multiple treatment groups receiving signals, where a signal is a piece of relevant information. Then, the experiment elicits features of agents' posterior beliefs, and records the actions taken under those posterior beliefs. Finally, the experiment uses group assignment to instrument for beliefs in two-stage least squares (TSLS) regressions of actions on posterior features: TSLS specifications in passive control experiments compare the control group to the treatment group(s), whereas specifications in active control experiments compare across treatment groups.
In Section (ref), we introduce an instrumental variables (IV) framework in which beliefs affect actions through features. We consider the partial effects of features on actions, which we allow to vary across agents and feature values. In Section (ref), we characterize TSLS estimators from the literature as weighted average partial effects (APEs) across these two margins. Notably, without further structure on belief updating to ensure IV monotonicity, the weights on agents' partial effects can be negative, which compromises the causal interpretation of TSLS estimators imbens1994identification, blandhol2022tsls.
In Section (ref), we propose conditions that ensure IV monotonicity. First, we propose signal monotonicity, which formalizes the idea that agents should update their beliefs “towards” the signals. We motive signal monotonicity in a model of posterior formation where agents perceive the signals as realizations from probability distributions that satisfy the monotone likelihood ratio (MLR) property. This allows for a broad class of signal distributions and belief updating rules. In particular, signal monotonicity nests existing models that assume Gaussian distributions and Bayesian updating armantier2016price, cavallo2017inflation, armona2019home, cullen2022much, fuster2022expectations, balla2022determinants. We additionally propose control-group stability for passive control comparisons, and treatment-group neutrality for active control comparisons; both formalize common experimental design considerations.
Under these conditions, TSLS estimators recover APEs that weight agents according to how much their beliefs respond to information provision. In particular, the weights for active control estimators emphasize agents with larger differences in their counterfactual posterior features across treatment groups (i.e., low versus high signal). Analogously, the weights for passive control estimators emphasize agents with larger differences in their counterfactual posterior features across treatment and control (i.e., signal versus no signal). However, some passive control estimators (i) up-weight agents with larger perception gaps (i.e., the difference between prior features and signals); or (ii) are contaminated by linear combinations of the TSLS first-stage coefficients. The latter allows for negative weights, which we advise against. The former ensures non-negative weights, but leads to a different APE than the baseline case above. We interpret these differences and give practical recommendations in Section (ref).
In Section (ref), we take our results on passive control estimation to kumar2023effect and jager2022worker. In the kumar2023effect application, we find that for three outcomes, the uncontaminated estimators produce coefficients that are one-third to two-thirds of the magnitude of those produced by the contaminated estimators; for one outcome, while the contaminated coefficients are statistically significant from the null, the uncontaminated coefficients are not. Together, we take this as evidence that negative weights meaningfully impact the estimates from the contaminated specifications. In the jager2022worker application, for one outcome, the non-up-weighted estimator produces a coefficient that is 30-40% larger in magnitude than those of the other specifications. Altogether, these empirical results highlight that the choice of weights can substantially impact the magnitude and significance of one's estimates.
\paragraph{Related Literature.} Our formulation of the causal effects of beliefs as the partial effects of features aligns with existing empirical practice, and nests models considered in the literature where agents optimize their actions as a function of their beliefs cullen2022much, balla2023identifying, jager2022worker. In allowing for heterogeneous partial effects, we are consistent with characterizations of IV estimators from other settings with continuous endogenous variables angrist2000interpretation, rambachan2021common, andrews2023causal. To our knowledge, heterogeneity in the feature value margin has not received explicit attention in the information provision literature---we show in Section (ref) that accounting for this margin affects the interpretation of the policy counterfactuals captured by passive and active control experiments.
Our proposed conditions correspond to notions from the IV literature. For active control estimators, signal monotonicity and treatment-group neutrality correspond to the canonical IV monotonicity condition imbens1994identification, angrist2000interpretation. In particular, agents' posterior features are larger when provided a high signal as opposed to a low signal. For passive control estimators, signal monotonicity and control-group stability correspond to “weak” IV monotonicity, which allows the direction of IV monotonicity to vary by covariates sloczynski2020should, blandhol2022tsls. In particular, agents' posterior features are larger when provided a signal above their prior feature, and smaller when provided a signal below their prior feature. Finally, the contamination that we find in some passive control estimators is an instance of TSLS specifications failing to be “monotonicity correct” in their first-stage blandhol2022tsls.
haaland2023designing survey applications of information provision experiments, and give guidance on experimental design, belief elicitation techniques, and other technical challenges. In contrast, we develop theory for the identification and estimation of causal effects. The closest to our paper in this regard is the independent and concurrent work of balla2023identifying, who likewise studies the interpretation of TSLS estimators in information provision experiments. balla2023identifying considers the partial effects of expectations, and targets an APE that places equal weights across agents. To identify this APE, balla2023identifying restricts heterogeneity in the feature value margin, and appeals to the structure of (i) active control comparisons; and (ii) linear updating of expectations. In contrast, since we primarily seek to characterize existing specifications, we achieve identification under weaker conditions on agents' actions, in both passive and active control experiments, and for more general learning environments. Finally, while balla2023identifying also interprets TSLS specifications from the literature, we discuss and characterize a more comprehensive set of specifications in a more general framework.
Let $\Delta(\Omega)$ denote the set of probability distributions over states $\omega \in \Omega$. Agents' beliefs $B$ are contained in a subset of distributions $\mathcal{B} \subseteq \Delta(\Omega)$. Examples include (i) sets of distributions for which relevant moments exist; and (ii) families of parametric distributions considered in the literature. We use $\phi: \mathcal{B} \to \mathbb{R}$ to index features of interest. For instance, when $\Omega \subseteq \mathbb{R}$, examples include the mean $\mu(B) := \int \omega dB(\omega)$ and the variance $ \sigma^{2}(B) := \int (\omega - \mu(B))^{2} dB(\omega)$.
An information provision experiment generally proceeds as follows. First, the experiment elicits features $\phi(B_{i0})$ of agents' prior beliefs $B_{i0}$. Next, the experiment randomly assigns agents to groups $g \in \mathcal{G}$. In passive control experiments, the groups are $\mathcal{G} = \{C, T\}$: Agents assigned to treatment receive signals $S_{i}^{T}$, whereas agents assigned to control receive no information $S_{i}^{C} := \varnothing$. In active control experiments, the groups are $\mathcal{G} = \{L, H\}$: Agents assigned to high treatment receive signals $S_{i}^{H}$, whereas agents assigned to low treatment receive signals $S_{i}^{L} < S_{i}^{H}$.
Given group assignment $G_{i} \in \mathcal{G}$, agents form posterior beliefs $B_{i1} \equiv B_{i1}^{G_{i}}$. These posterior beliefs influence agents' actions/outcomes $Y_{i} \in \mathbb{R}$, which the experiment records; we assume there exist functions $Y_{i}^{g}(B)$ such that $Y_{i} \equiv Y_{i}^{G_{i}}(B_{i1})$. Finally, the experiment elicits posterior features $\phi(B_{i1})$. The goal is to estimate the causal effects of beliefs on actions.
The random assignment of agents to groups provides a basis for estimating causal effects. Formally, we assume that group assignment $G_{i}$ is a valid instrument. In what follows, $X_{i}$ is a vector of agent characteristics.
Independence means that the experiment cannot condition group assignment $G_{i}$ on agent characteristics $X_{i}$, prior features $\phi(B_{i0})$, anticipated actions, and so on. However, it allows the content of the signals $S_{i}^{g}$ to depend on such variables jager2022worker, deshpande2023lack. In any case, random assignment is sufficient for Assumption (ref)(i).
Exclusion means that group assignment only impacts actions through posterior beliefs. Therefore, one practical concern is that information provision may also affect actions through emotional responses haaland2023designing. However, Assumption (ref)(ii) does accommodate emotional responses that solely affect posterior formation---examples include belief-based utility or motivated reasoning brunnermeier2005optimal, epley2016mechanics.
To formalize a tractable notion for the causal effects of beliefs on actions, we assume that actions depend on beliefs through a single feature of interest $\phi$. In what follows, we suppose that the set of possible feature values $\phi(\mathcal{B}) := \{\phi(B): B \in \mathcal{B}\}$ is convex: For any two values in $\phi(\mathcal{B})$, any third value between them can be rationalized by some $B \in \mathcal{B}$. This ensures that the causal effects defined in Section (ref) correspond to beliefs that the agents could actually hold.\footnote{Convexity holds whenever the set of possible beliefs $\mathcal{B}$ is sufficiently rich---see Appendix (ref).}
Assumption (ref) aligns with existing practice from the literature, which often frames the causal effects of beliefs on actions in terms of features. If $\mathcal{B}$ is parametrized by $\phi$, then actions depend on beliefs through $\phi$; examples include one-parameter exponential families lehmann2006theory. If we broaden $\mathcal{B}$ to be the set of distributions with finite second moments, then another approach is to restrict preferences. For example, if we assume that agents are risk neutral in the sense that their optimal actions only depend on beliefs via first moments, then $Y_{i}(B) = Y_{i}^{\mu}(\mu(B))$. We formalize these arguments in Appendix (ref).
In some cases, Assumption (ref) allows beliefs to affect actions through additional features. In particular, if $Y_{i}(B) = Y_{i}^{\phi, \eta}(\phi(B), \eta(B))$ and $\eta(B_{i1}^{g}) = \eta(B_{i1}^{\Tilde{g}})$ for $\mathcal{G} = \{g,\Tilde{g}\}$, then we can take $Y_{i}^{\phi}(\phi(B)) := Y_{i}^{\phi, \eta}(\phi(B), \eta(B_{i1}^{g}))$. This accommodates models from the literature that predict $\sigma^{2}(B_{i1}^{H}) = \sigma^{2}(B_{i1}^{L})$ in active control experiments with $\phi = \mu$---see Appendix (ref). An analogous argument applies for cross-learning, which is when information about one state affects beliefs about another state haaland2023designing. For example, information about inflation may influence expectations of both inflation and economic growth coibion2023forward. In such cases, it suffices to rule out differences in cross-learning across $\{g,\Tilde{g}\}$.
One caveat is that, given a pair of comparison groups, we cannot allow multiple features to change at the same time. For example, we rule out comparisons where agents in one of the groups receive multiple signals with the intention of shifting multiple features cullen2022much, kumar2023effect, coibion2021effect. Otherwise, there would be multiple endogenous variables; in such cases, TSLS estimators generally do not recover interpretable causal parameters bhuller20222sls.
We formulate the causal effects of beliefs on actions as the partial effects $\partial_{v}Y_{i}^{\phi}(v)$ of feature $\phi$.\footnote{In Appendix (ref), we discuss alternative formulations, including one that accomodates discrete actions.} TSLS recovers APEs across feature values $v$ and agents $i$. In Sections (ref) and (ref), we derive these APEs for passive and active control specifications from the literature. In Section (ref), we summarize key takeaways and motivate the need for assumptions on belief updating behavior.
In what follows, $W_{i}$ is a vector that includes $1$ and potentially other variables that are independent of $G_{i}$. Let $\textnormal{sign}(v) := \mathds{1}\{v \geq 0\} - \mathds{1}\{v \leq 0\}$ and $\psi_{i}^{g\tilde{g}} := \textnormal{sign}(\phi(B_{i1}^{\tilde{g}}) - \phi(B_{i1}^{g}))$. Let $\lambda_{i}^{g\tilde{g}}(v)$ denote the density function of the uniform distribution on the values $v$ between $\phi(B_{i1}^{g})$ and $\phi(B_{i1}^{\tilde{g}})$. Propositions (ref) and (ref) follow from TSLS algebra and the fundamental theorem of calculus, in the spirit of angrist2000interpretation. See Appendix (ref) for details.
Passive control specifications recover an APE, constructed as follows. First, for each agent $i$, the partial effect curve $ \partial_{v}Y_{i}^{\phi}(v)$ is aggregated across the range of values $v$ between $\phi(B_{i1}^{C})$ and $\phi(B_{i1}^{T})$. The density function $\lambda_{i}^{CT}(v)$ places uniform weight across this range of feature values, which generates $\Bar{\beta}_{i}^{CT}$. Then, $\Bar{\beta}_{i}^{CT}$ is averaged across agents with weights $w_{i}^{CT}$ that are proportional to (i) the signed difference $|\phi(B_{i1}^{T}) - \phi(B_{i1}^{C})|\psi_{i}^{CT}$ in $i$'s counterfactual posterior features across treatment and control; and (ii) the agent-specific linear combination $I_{i}'\pi$ of first-stage coefficients:
The weights $\lambda_{i}^{CT}(v)$ for the feature values are non-negative and integrate to one. The weights $w_{i}^{CT}$ for the agents integrate to one, but can be negative due to $\psi_{i}^{CT}$ and $I_{i}$. The literature considers a number of “interactions” $I_{i}$---note that $\pi$ cancels out when $I_{i}$ is a scalar.
Active control specifications recover an APE that is constructed analogously to the passive control case. The weights $\lambda_{i}^{LH}(v)$ for the feature values are non-negative and integrate to one. The weights $w_{i}^{LH}$ for the agents integrate to one, but can be negative due to $\psi_{i}^{LH}$.
Thus far, we assumed that (i) group assignment is a valid instrument for posterior beliefs; and (ii) actions depend on beliefs through a single feature of interest. Under these assumptions, we showed that TSLS specifications from the literature recover APEs $\beta^{CT}$ and $\beta^{LH}$ with weights $w_{i}^{CT}$ and $w_{i}^{LH}$ that can be negative. This negative weighting compromises the causal interpretation of $\beta^{CT}$ and $\beta^{LH}$. For instance, if some of the weights are negative, then it is possible for $\beta^{CT}$ and $\beta^{LH}$ to be negative even when all partial effects are positive blandhol2022tsls. To address this concern, we now place structure on belief updating.
The weights $w_{i}^{g\tilde{g}}$ are contaminated by $\psi_{i}^{g\tilde{g}} := \textnormal{sign}(\phi(B_{i1}^{\tilde{g}}) - \phi(B_{i1}^{g}))$. This source of contamination suggests that we should restrict the “direction” of agents' responses to information provision. Section (ref) introduces a signal monotonicity condition that makes this idea precise. To achieve IV monotonicity for passive control, Section (ref) additionally introduces a control-group stability condition---Section (ref) then characterizes $w_{i}^{CT}$. To achieve IV monotonicity for active control, Section (ref) additionally introduces a treatment-group neutrality condition---Section (ref) then characterizes $w_{i}^{LH}$.
For agent $i$ in group $g$, let $B_{i1}^{g}(\cdot|s)$ be the belief updating rule, which maps signals $s \in \mathbb{R} \cup \varnothing$ to beliefs $B \in \mathcal{B}$. Note that $B_{i1}^{g}(\cdot|s)$ implicitly depends on priors $B_{i0}$, and $B_{i1}^{g} \equiv B_{i1}^{g}(\cdot|S_{i}^{g})$ in the realized experiment.
Assumption (ref) restricts the direction of agents' responses to information provision; intuitively, agents should update “towards” the signal. This assumption is reasonable in many experiments. For example, in the setting of kumar2023effect, firms in one treatment group are given the average forecasts of mean GDP from a panel of experts. It is reasonable to assume that larger average forecasts lead firms to form higher expectations of mean GDP. In another treatment group, firms are given the difference in forecasts between the most and least optimistic experts. It is reasonable to assume that larger dispersion in forecasts leads firms to be more uncertain in their beliefs.
However, Assumption (ref) can sometimes be invalid. For example, consider coibion2022monetary, who study households' expectations of inflation. In one treatment group, they provide information about unemployment. Some households may believe that higher unemployment implies lower inflation, whereas others may believe the opposite. Therefore, signal monotonicity is unreasonable for this treatment group.
As demonstrated, we can often intuit the validity of signal monotonicity. To complement this intuition, Appendix (ref) formalizes conditions under which Assumption (ref) is satisfied for the leading case of $\phi = \mu$. In summary, the main condition is that agents perceive the signals as realizations from distributions that satisfy a monotone likelihood ratio (MLR) property. In particular, agents assigned to treatment assume that the experiment tends to provide larger signals under larger realizations of the state.
The MLR property is satisfied for the set of Gaussian beliefs considered in the information provision literature---see Appendix (ref). More generally, many exponential families satisfy the MLR property in their respective sufficient statistics casella2021statistical. Given the MLR structure on agents' perceptions, we show in Appendix (ref) that Assumption (ref) is satisfied for $\phi = \mu$ under various belief updating rules, including (i) the Bayesian baseline considered in the information provision literature armantier2016price, cavallo2017inflation, armona2019home, cullen2022much, fuster2022expectations, balla2022determinants; and (ii) systematic deviations from Bayesian updating considered in the behavioral economics literature gabaix2019behavioral, benjamin2019errors.
We first consider passive control comparisons. Let $S_{i}^{\phi} := \phi(B_{i0})$ denote agent $i$'s prior feature.
If a passive control comparison is control-group stable, then the provision of $s = S_{i}^{\phi}$ in the treatment group is the same as not providing any information in the control group. Notice
Thus, under control-group stability, signal monotonicity implies
In particular, with an appropriate TSLS specification, we can use the sign of the perception gap to correct for the contamination from $\psi_{i}^{CT}$. Thus, signal monotonicity and control-group stability correspond to “weak” IV monotonicity, which allows the direction of canonical IV monotonicity to vary across covariate values sloczynski2020should, blandhol2022tsls.
The stability condition assumes that receiving information “consistent” with one's prior feature is equivalent to not receiving any information, in the sense that the posterior features are the same. For example, consider an agent that (i) has prior expectations of 3% for inflation; and (ii) without additional information, maintains their prior beliefs. Stability assumes that this agent's posterior expectations remain at 3% if given a signal that inflation will be 3%.\footnote{Here it is fine if the agent's uncertainty decreases---we only need expectations to be stable.}
The stability condition restricts the signal to be directly comparable to the state of interest. For example, consider a design in which the elicited priors and posteriors are expectations $\mu$ over the extent of discrimination against Black individuals in the housing market, and the signal measures the extent of discrimination against Black individuals in the labor market. This treatment group plausibly satisfies signal monotonicity: It is reasonable that agents believe that higher levels discrimination against Black individuals in the labor market tend to imply higher levels of discrimination in the housing market. However, the value of $\mu(B_{i0})$ refers to a different state (labor market) than $S_{i}^{T}$ (housing market). In particular, being told that the level of discrimination in the housing market is $S_{i}^{\mu} := \mu(B_{i0})$ need not confirm one's priors over that discrimination.\footnote{An alternative design is to elicit prior expectations over the state corresponding to the signal, as is done in haaland2023racial, who estimate an intent-to-treat effect. Our framework considers a common state space $\Omega$ for the prior and posterior for simplicity---to our knowledge, no existing paper that provides TSLS estimates uses the alternative design in haaland2023racial.}
The stability condition limits the extent to which agents may update their beliefs beyond the signal provided, between the elicitation of priors and posteriors. For example, if agent $i$ searches for information with greater intensity when assigned to control than when assigned to treatment with a signal $s = S_{i}^{\phi}$ that “confirms” their priors, then $\phi(B_{i1}^{T}(\cdot|S^{\phi}_{i})) \neq \phi(B_{i1}^{C})$. However, in such cases it suffices to assume a weaker version of stability:
This allows agents to acquire outside information cavallo2017inflation, armona2019home, provided that the difference in $\phi$ from treatment with $s = S_{i}^{T}$ versus $s = S_{i}^{\phi}$ is large enough.
Proposition (ref), which follows from the analysis in Section (ref), characterizes $w_{i}^{CT}$.
Thus, under signal monotonicity and control-group stability, only some of the passive control specifications from the literature recover positive-weighted APEs. The weights from $I_{i}^{sign}$ are positive, and proportional to the absolute difference in $i$'s counterfactual posterior features across treatment and control.\footnote{If $S_{i}^{T} = \phi(B_{i0})$ for some $i$, then the weights for $I_{i}^{sign}$ are $w_{i}^{CT} \propto |\phi(B_{i1}^{T}) - \phi(B_{i1}^{C})|\mathds{1}\{S_{i}^{T} \neq \phi(B_{i0})\}$.} Relative to $I_{i}^{sign}$, interaction $I_{i}^{gap}$ additionally up-weights agents with larger absolute perception gaps. Relative to $I_{i}^{sign}$ and $I_{i}^{gap}$, interactions $I_{i}^{1,gap}$ and $I_{i}^{1,prior}$ allow for negative weights. Intuitively, the signs of the scalars $I_{i}^{sign}$ and $I_{i}^{gap}$ correctly predict $\psi_{i}^{CT}$. On the other hand, the vectors $I_{i}^{1,gap}$ and $I_{i}^{1,prior}$ induce linear combinations $I_{i}'\pi$ that need not correctly predict $\psi_{i}^{CT}$.
We now consider active control comparisons.
If an active control comparison is treatment-group neutral, then the provision of $s = S_{i}^{L}$ in both treatment groups leads agent $i$ to form the same posterior features. Notice
Thus, under treatment-group neutrality, signal monotonicity implies
In particular, we can use $\textnormal{sign}(S^{H}_{i} - S^{L}_{i})$ to correct for the contamination from $\psi_{i}^{LH}$. However, since $S_i^H > S_i^L$ for all $i$, then $\textnormal{sign}(S^{H}_{i} - S^{L}_{i}) = 1$ so that existing active control specifications “automatically” make the correction. Therefore, signal monotonicity and treatment-group neutrality correspond to the canonical IV monotonicity condition imbens1994identification, angrist2000interpretation.
Intuitively, the neutrality condition assumes that agents perceive the provision of information to be “equally credible” across treatment groups.\footnote{It suffices to assume that the belief updating rules $B_{i1}^{g}(\cdot|s)$ are the same across $g \in \{L, H\}$.} This equality may not hold when there are asymmetric effects of information provision across treatment groups. For example, agents may perceive information provision to be less credible in groups that provide information that more strongly challenges their prior beliefs gentzkow2006, haaland2023designing. Such issues can be mitigated with appropriate experimental design, including framing the information as coming from similar sources and using the same presentation strategies across treatment groups. Furthermore, as with control-group stability, we can also entertain a weaker version of neutrality:
This allows for some perceived asymmetry, provided that the difference in $\phi$ from treatment with $s = S_{i}^{H}$ versus $s = S_{i}^{L}$ is large enough.
Proposition (ref), which follows from the analysis in Section (ref), characterizes $w_{i}^{LH}$.
In particular, under signal monotonicity and treatment-group neutrality, existing active control specifications recover positive-weighted APEs. The weights $w_{i}^{LH}$ are proportional to the absolute difference in agents' posterior features across treatment groups, which parallels $w_{i}^{CT}$ for the passive control specification with interaction $I_{i}^{sign}$.
Given our framework, when deciding between active and passive control designs, researchers should consider how the associated comparisons $\phi(B_{i1}^{H}) - \phi(B_{i1}^{L})$ and $\phi(B_{i1}^{T}) - \phi(B_{i1}^{C})$ differ in (i) the causal parameters they recover; and (ii) the identification constraints they place on the design of information provision.
Within a given agent $i$, passive control aggregates the partial effects $\partial_{v}Y_{i}^{\phi}(v)$ over the range of values $v$ between $\phi(B_{i1}^C)$ and $\phi(B_{i1}^T)$, directly answering the question “what is the effect of giving agent $i$ information, relative to not giving them information?” In contrast, active control aggregates over values $v$ between $\phi(B_{i1}^L)$ and $\phi(B_{i1}^H)$, answering the question “what is the effect of giving agent $i$ one piece of information relative to another?” The latter counterfactual may have less policy relevance, as noted in haaland2023designing.
Notably, even if $|\phi(B_{i1}^{T}) - \phi(B_{i1}^{C})| = |\phi(B_{i1}^{H}) - \phi(B_{i1}^{L})|$, we can still have $\beta^{CT} \neq \beta^{LH}$. For a simple illustration of this difference, let us assume that $S_i^T = S_i^H$, such that $B_{i1}^T = B_{i1}^C = B_{i1}^*$. If we trace out a partial effects curve $\partial_{v}Y_{i}^{\phi}(v)$ as in Figure (ref), then $\Bar{\beta}^{LH}_{i}$ is proportional to the green region and $\Bar{\beta}^{CT}_{i}$ is proportional to the orange region. Due to the shape of the partial effects curve for this agent, $\Bar{\beta}^{CT}_{i} > \Bar{\beta}^{LH}_{i}$. Thus, even if $|\phi(B_{i1}^{T}) - \phi(B_{i1}^{C})| = |\phi(B_{i1}^{H}) - \phi(B_{i1}^{L})|$ for all $i$, we can still have $\beta^{CT} \neq \beta^{LH}$.
Passive and active control also differ in how they aggregate partial effects across agents. The latter weights agents according to $|\phi(B_{i1}^H) - \phi(B_{i1}^L)|$, while the former with interaction $I_i^{sign}$ weights agents according to $|\phi(B_{i1}^T) - \phi(B_{i1}^C)|$.\footnote{We focus on $I_i^{sign}$ because it induces weights $w_{i}^{CT}$ that are most analogous to $w_{i}^{LH}$.} Because agents assigned to control are not given information, $|\phi(B_{i1}^T) - \phi(B_{i1}^C)|$ is more sensitive to agents' priors. When $|\phi(B_{i1}^T) - \phi(B_{i1}^C)|$ is more heterogeneous than $|\phi(B_{i1}^H) - \phi(B_{i1}^L)|$, passive control places less equal weights across agents; this is more likely in settings with highly heterogeneous priors---see Appendix (ref). Therefore, although active control generates a potentially less policy-relevant counterfactual, it may place more equal weights across agents, which may be desirable in some settings balla2023identifying. This analysis informs discussions from roth2020expectations and haaland2023designing, which posit that active control identifies causal effects for a “broader population” of agents. This claim is not exactly true; rather, it is the relative weight on agents with differing priors that active control may better equalize.
Finally, active control requires treatment-group neutrality, whereas passive control requires control-group stability. Under neutrality, $S_i^L$ and $S_i^H$ must be perceived to be equally credible; this imposes design constraints on how distinct the two signals can be (since more extreme signals may lead to loss of credibility), and what sources may be used to generate the signals. Under stability, $S_i^T$ must convey similar numerical content to the prior feature (i.e., same units for the signals as the prior features), and agents should not differentially search for additional information between prior and posterior elicitation, dependent on group assignment.
Among the four passive control specifications, interactions $I_i^{sign}$ and $I_i^{gap}$ generate positive weights, whereas $I_i^{1,gap}$ and $I_i^{1,prior}$ do not. Therefore, the latter two specifications should not be used: Agents' partial effects may enter negatively into the weighted average, attenuating or reversing the sign of the estimated APE.
Between $I_i^{sign}$ and $I_i^{gap}$, the former directly maps to canonical formulations of the local average treatment effect for continuous endogenous variables angrist2000interpretation, which weights by the size of agents' “first-stages.” In this setting, these agent first-stages are given by $|\phi(B_{i1}^{T}) - \phi(B_{i1}^{C})|$. However, if researchers are especially interested in agents with large perception gaps, then $I_i^{gap}$ may generate a more useful APE. Moreover, $I_i^{sign}$ and $I_i^{gap}$ may provide different levels of estimation precision.
We now compare the different passive control interactions using data from kumar2023effect and jager2022worker. We show that the chosen specification affects the magnitude and significance of estimates, and provide evidence that these effects are driven by contaminated weights in the former setting, and by up-weighting of agents with larger perception gaps in the latter setting.
kumar2023effect study how firms' expectations and uncertainty of future GDP growth affect their economic decisions. We compare two arms in their experiment: the control arm, which receives no information, and their first treatment arm, which tells firms that the average prediction of GDP growth among a panel of professional forecasters is 4%. kumar2023effect elicit agents' expectations and uncertainty---we focus on expectations.\footnote{As noted in Section (ref), econometric issues arise when there is more than one endogenous variable; hence, we avoid including both expectations and uncertainty.} The outcome variable $Y_{i}$ is defined as the difference between realized and planned changes in business choices, such as prices and employment.
Figure (ref) plots TSLS coefficients for various outcomes: price, employment, and advertising budget. Other outcomes are given in Appendix (ref). Since all firms in the treatment group receive the same professional forecast, interactions $I_i^{1,gap}$ and $I_i^{1,prior}$ provide the same estimates.
For price, the magnitude of the coefficient for interaction $I_i^{sign}$ is halved and the magnitude of the coefficient for interaction $I_i^{gap}$ is reduced by one-third, relative to the coefficients for interactions $I_i^{1,gap}$ and $I_i^{1,prior}$. A t-test of these coefficients finds that $I_i^{sign}$ is marginally significantly different (p-value = 0.078) from $I_i^{1,gap}$ and $I_i^{1,prior}$. Similarly, for employment, the magnitude of the coefficients for $I_i^{sign}$ and $I_i^{gap}$ are 60% and 50% of the other coefficients; $I_i^{gap}$ is marginally significantly different (p-value = 0.081) from $I_i^{1,gap}$ and $I_i^{1,prior}$. Finally, for advertising budget, the magnitudes of the coefficients for $I_i^{sign}$ and $I_i^{gap}$ are a third of the coefficients for interactions $I_i^{1,gap}$ and $I_i^{1,prior}$. In fact, under $I_i^{sign}$ and $I_i^{gap}$, the 95% confidence interval on the advertising budget estimate include zero effects.
What explains the above differences? Recall that interactions $I_i^{sign}$ and $I_i^{gap}$, which estimate coefficients of similar magnitude, both generate positive weights. In contrast, interactions $I_i^{1,gap}$ and $I_i^{1,prior}$, which estimate coefficients that are twice as large, both allow for negative weights. Although we cannot formally reject the non-existence of negative weights for interactions $I_i^{1,gap}$ and $I_i^{1,prior}$, the above grouping pattern suggests that negative weights may be driving larger TSLS estimates for $I_i^{1,gap}$ and $I_i^{1,prior}$.
jager2022worker study how information about workers' outside options---that is, workers' wages if they left their current job to find a new one---affects their labor market decisions. First, jager2022worker elicit workers' prior expectations of their outside options. Next, they split their sample into a control group and a single treatment group. The control arm receives no information; those in the treatment group are told the mean wage of workers similar to themselves (based on gender, age, occupation, labor market region, and education level). Then, jager2022worker elicit workers' posterior expectations of their outside options.
Figure (ref) displays estimated coefficients from all four interactions using intended negotiation probability as the outcome---estimates for other outcomes are in Appendix (ref). The interaction $I_i^{sign}$ coefficient is 30-40% larger (though not statistically significantly so) than the coefficients from interactions $I_i^{gap}$, $I_i^{1,gap}$, and $I_i^{1,prior}$, each of which are similar to each other. The fact that interaction $I_i^{gap}$, which ensures positive weights, is similar to interactions $I_i^{1,gap}$ and $I_i^{1,prior}$ suggests that the potential negative weighting in the latter is not consequential in this setting. Instead, the smaller coefficients for interactions $I_i^{gap}$, $I_i^{1,gap}$, and $I_i^{1,prior}$ may be due to the up-weighting of agents with larger perception gaps. We provide further evidence that up-weighting may be driving these results in Appendix (ref).
One explanation for why up-weighting may have attenuated the estimates is endogenous information acquisition, such as in balla2023identifying. Agents who are more responsive to information are more likely to seek it out, and therefore have more accurate priors. At the same time, agents that are very accurate will not respond much to the information treatment; their actions are already near-optimal. Therefore, the agents that are most responsive are those that have medium-sized perception gaps, and the agents that are least responsive are those that have large or no perception gap. Interactions $I_i^{gap}$, $I_i^{1,gap}$, and $I_i^{1,prior}$ will up-weight agents with large perception gaps---and thus those who are least responsive---attenuating treatment effects towards zero.
This paper examines identification and estimation in information provision experiments. We developed a causal framework for these experiments, and provided conditions for recovering interpretable APEs in both passive and active control designs. Using this framework, we evaluated TSLS specifications from the literature. We found that the standard active control specification identifies an APE with non-negative weights. However, two of the four common passive control specifications allow for negative weights, which we advise against.