Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
42,139 characters · 0 sections · 54 citation commands
JEL: C81, C93, D74, P00\\ Keywords: Belief Elicitation, Experimental Designs, Identification, Political Protests
\@startsection{section}{1} \z@{1.0\linespacing\@plus\linespacing}{.8\linespacing}{Introduction}
Economic theory posits that an individual's actions in a strategic environment depend on their preferences over outcomes and their beliefs about how others would act. However, measuring the causal role of beliefs on actions from observational data is inherently difficult. After all, even when actions are observable, beliefs are rarely observed. Meanwhile, imputing beliefs from actions requires presupposing their causal effect, even though that is often the empirical goal itself.
Experimental methods provide an attractive solution in these contexts. In experiments, researchers can observe actions and elicit beliefs, both before and after carefully designed information interventions. For example, within the context of political activism, cyyz, jarke2021free, hager2022group, hager2022political all elicit subjects' beliefs about others' planned participation in political events/protests. They then measure how changing subjects' information about others' participation causally affects individual beliefs and, thence, individuals' own attendance.\footnote{Such examples go beyond political economy. In another recent example, bursztyn evaluate whether Saudi husbands are more likely to support women working outside the home if they discover that other husbands do so too.}
However, eliciting beliefs is fundamentally different than eliciting simpler variables, such as willingness-to-pay. This is because beliefs are a probability distribution, often defined over a large set of outcomes, rather than just a point response (i.e., a real number between 0 and 1). For example, the belief about the proportion of $N$ other survey-participants who participate in a protest is a probability distribution over the $N+1$ possible values $\{0,1/N, 2/N,..1\}$. Yet, for tractability, researchers often have to elicit point-responses in their experimental design, and interpret those as a coarse measure of beliefs. This is the case in the papers cited above, as well as many others.\footnote{trebbi, trebbi2 are some exceptions in the experimental political economy context.}
In this paper, we discuss the relation between subjects' point-responses and their underlying belief distribution, and how this mapping depends on incentives offered to the subject. In particular, we compare two popular belief elicitation schemes that seem superficially similar, but, as we show in Section (ref), incentivize different best responses in belief reporting. We then show the empirical consequences of such differences, which can include identifying effects with opposite signs to the true ones, or a lack of identification altogether.
The first scheme we consider rewards subjects for correct guesses within an error band of the true value: for instance, “Please guess $x\in [0,1]$. If your guess is within $\Delta$ percentage points of $x\in [0,1]$ you will earn a bonus payment of 1 currency unit." Recent examples of belief elicitation using this scheme include cyyz, chenyang, bursztyn, among others. In Section (ref), we prove that subjects' best response to these incentives is to report the (approximate) mode of the true distribution of $x$ (henceforth, modal beliefs). The second one, which we recommend to practitioners who wish to elicit the mean of the belief distribution over $x$ (mean beliefs, henceforth), rewards subjects $A-B(x-r)^2$ for reporting $r$, where $A,B$ are constants. Such an incentive scheme indeed induces profit-maximizing subjects to report their mean beliefs as a best response.
The difference in elicitation designs is subtle: rewards under both schemes are weakly increasing in accuracy. But they induce very different mappings from the true belief distribution (over $x$) to the optimal report, as mean and mode do not generally coincide. For the sake of completeness, we also discuss a third incentive scheme, which rewards subjects based on the absolute distance between their report and the true value. This incentivizes reporting the median of the true distribution instead.
We show that these differences can be very consequential when it comes to testing a theoretical prediction. Not only are modes, means and medians generally different, but they may have very different properties following an information intervention. This can be visualized within a simple example, shown in Figure (ref). In this example, beliefs about a random variable, $x$, are assumed to follow a flexible continuous distribution, which has been calibrated to our empirical application (described below). In Figure (ref), the mode is given by 0.142, while the mean is 0.273. Now, we assume there is an experimental information intervention, like those in the papers cited above. In particular, the treatment reveals that $x$ is equal to 0.17, akin to cyyz. Beliefs will then update towards 0.17, implying the mode will increase towards 0.17, while the mean will decrease accordingly, as illustrated by the arrows in the figure.
Hence, a researcher who uses incentives to measure modal beliefs but interprets the data as mean beliefs, will have identified a very different measure than they intended to. And using this measure may lead to very different conclusions while testing a theory-derived hypothesis. In fact, as the example above shows, researchers who interpret modal and mean beliefs interchangeably, may (i) fail to identify the theoretical parameter of interest, or may (ii) identify their opposite sign (if the parameter is defined by how changes in mean beliefs relate to changes in actions). We formalize these points in Section (ref).
As an empirical application, we revisit cyyz's experiment on political protests in Hong Kong.\footnote{We also discuss hager2022group, jarke2021free, hager2022political that use a similar research design and where our insights also apply.} cyyz conduct a cleverly designed field experiment in the backdrop of an anti-authoritarian protest in Hong Kong. They show that potential protesters who increased their reported beliefs in favor of others being more likely to protest, became less likely to protest themselves. Interpreting the reported beliefs as mean beliefs, cyyz conclude that “specifically, our findings reject many recent models that assume only the possibility of strategic complementarity in the protest decision." (p.1072) That is, they interpret their evidence as contradicting the widespread assumption that political protests are strategic complements (i.e., when a citizen believes that more peers are likely to protest, she should become more likely to protest herself, see gehlbach).
However, we argue that the empirical results in cyyz do not necessarily reject strategic complementarity (SC) or the many theoretical models that assume it. Rather, we show that cyyz's evidence can be completely consistent with SC using a two-fold argument. First, cyyz's experimental design uses the first belief-elicitation scheme described above, thereby incentivizing subjects to report their modal beliefs, but not their mean beliefs. Thus, their belief data does not necessarily reveal mean beliefs. Second, we use a stylized theoretical model and closely derived statistical results to show that means and modes can be updated in opposite directions in their set-up, as shown in Figure (ref). Therefore, their estimates of a negative covariance between belief updating about others' protest attendance and one's own attendance may be flipped.
The distinction borne in our discussion is also empirically meaningful. We perform a numerical exercise where we fit flexible distributions of individual beliefs to the distribution of observed prior and posterior reports in the cyyz dataset, assuming subjects were reporting modal beliefs about others' participation. Based on the estimated distribution of beliefs, we find that over a third of subjects would have updated their mean belief in the opposite direction of their reported belief. Therefore, how one interprets “elicited beliefs” determines what the data of cyyz implies about strategic complementarity.
To be clear, our analysis does not aim to refute or reverse any regression results from cyyz. Rather, we use it as a prominent example of how elicitation schemes influence the interpretation of economic data: The scheme used in cyyz allows alternative interpretations of their results, and such interpretations are meaningful to what we can learn about the empirical validity of strategic complementarity. Such alternative explanations could have been completely avoided with the alternative mean-eliciting scheme that we propose. None of what we say precludes the possibility that subjects have unimodal symmetric beliefs and, thus, their mean and modal beliefs are identical, thereby preserving their original interpretation in cyyz. Instead, we think of our results as clarifying the important role that this implicit assumption (unimodal symmetric beliefs) plays in linking the regression results to the final conclusion, both in their experiment and in others, emphasizing the importance of the design of belief elicitation procedures per se.
\@startsection{section}{1} \z@{1.0\linespacing\@plus\linespacing}{.8\linespacing}{Belief Elicitation}
Suppose subjects believe that some outcome $x\in[0,1]$ is distributed according to $f$, a probability density function over $[0,1]$. In cyyz and hager2022group, for instance, $x$ is the percentage of the participants from the study who plan to participate in a political protest. In bursztyn, it is the percentage of Saudi husbands who believe women should be allowed to work outside the home. Suppose researchers are interested in eliciting a single value that is in some way representative of the distribution $f$.
\@startsection{subsection}{2} \z@{.8\linespacing\@plus.7\linespacing}{.7\linespacing}{Elicitation Schemes that Incentivize Reporting the Mode}
Consider the following general incentive system:
cyyz use a special case of this with $\Delta=2$, and where 1 currency unit was 10 HKD. Similarly, chenyang sets $\Delta = 0.1$ and the currency unit was RMB 5. Meanwhile bursztyn uses some $\Delta$ close to 0 with strategic uncertainty about its value and US\$20 as a currency unit.\footnote{Technically, the incentives in bursztyn award the participant “who guesses most accurately". This could induce strategic guessing, where a subject anticipates other subjects' guesses when choosing their report. However, with a large enough population of independent subjects (1500 in their sample), payment should reflect being arbitrarily close to the true values ($\Delta \to 0$).}
There exists simple sufficient conditions on $f$ under which, under this elicitation scheme, profit maximizing subjects should report the mode exactly, or approximately with an error band of $\Delta$ percentage points. If $\Delta = 0$ then the mode will trivially be the best response. The interesting case is when $\Delta>0$.
Part (i) provides a very simple sufficient condition under which subjects would exactly report the mode when beliefs are discrete. We will use (i) in a stylized model in Subsection (ref) to show that, following the information-treatment, the mode of $P_{-i}$ can be updated in the opposite direction to the mean of $P_{-i}$.
Part (ii) then characterizes the optimal report for the simple case of a continuous distribution where the density function has exactly one peak at the mode. In this case, the subject will (approximately) report the mode. Many density functions are single-peaked. For example, the $Beta$ distribution $f(x;\alpha,\beta)$ is single-peaked for any $\alpha,\beta>1$, a property we show formally in Claim (ref) in Appendix and use in Subsection (ref). Part (iii) proves the same result for the more general case where the density function is only locally single-peaked around the mode.
As a result, the papers cited above which elicit subjects using scheme (ref) will obtain reports of subjects' modal beliefs about others' behavior (e.g., protest attendance, opinions on women's work rights, etc.), rather than their mean beliefs.
\@startsection{subsection}{2} \z@{.8\linespacing\@plus.7\linespacing}{.7\linespacing}{Elicitation Schemes that Incentivize Reporting the Mean}
We now consider an alternative scheme that would elicit the mean instead, before discussing why these differences matter for empirical research.
Suppose subjects who reported $r$ are paid $B-A(x-r)^{2}$ if $x \in [0,1]$ is the true/realized value from $f$. $B$ and $A$ are constants chosen by the experimenter with $B>A>0$.
The proof is based on the mean being the minimizer of Mean Squared Error. Though the proof treats $f(\cdot)$ as a continuous distribution, a very similar proof would work even if $f(\cdot)$ were discrete.
Notice that both incentive schemes presented so far are based on the same concept: they reward subjects depending on how close their guesses are to the true realization of the random variable, $x$. However, the details of those incentives matter: whether the reward scheme is based on a quadratic loss function, or on a near-enough value can elicit completely different reports. Beyond unimodal symmetric distributions, modes and means do not coincide. This can affect both the magnitude and the sign of the conclusions drawn from such experiments, as we explain in Section (ref). Before that, though, we comment on two last, but relevant, elicitation schemes.
\@startsection{subsection}{2} \z@{.8\linespacing\@plus.7\linespacing}{.7\linespacing}{Elicitation Schemes that Incentivize Reporting the Median}
To reinforce our point about how incentive schemes matter, consider a slightly different incentive scheme where subjects who reported $r$ are paid $B-A|x-r|$ if $x \in [0,1]$ is the true proportion of participants' outcomes, and $B$ and $A$ are constants chosen by the experimenter with $B>A>0$. Even this small change to the reward function (from square to absolute value) ends up eliciting a different point of the distribution, in this case, the median. We show this separately for discrete and continuous distributions as a discrete distribution allows multiple medians.
\@startsection{subsection}{2} \z@{.8\linespacing\@plus.7\linespacing}{.7\linespacing}{Experimental Designs Without Incentivized Reporting}
Certain experimental designs do not provide any incentives for subjects when they report their beliefs about others' protest attendance. For example, hager2022group simply ask “How many protesters do you think will attend the AfD protest?" and “how many people will participate in the respective protests?", without rewards for correct guesses. Similar questions are found in jarke2021free, hager2022political, who do not reward subjects for correct responses either. The authors explain that incentives could “undermine this code [of honor for truthful answers] and make non-truthfulness salient in the first place" (jarke2021free) and that “the campaign were concerned that incentives would be perceived as very unusual by their supporters." (hager2022political).
While such reservations from institutional partners are natural, and while we understand that economists are forced to accept such constraints when working in the field, we also want to draw attention to the following point: Without incentives, it is no longer possible to map a subject's point-reports to their belief distribution. Recall our setting where subjects believe that some outcome $x\in[0,1]$ is distributed according to $f$. Absent any rewards, subjects' best responses to the elicitation questions above are not well-defined: Even rational subjects could be reporting the mode (as in (ref)), the mean (as in (ref)), the median (as in (ref)), or possibly, any other values within their support of beliefs. Perhaps, if beliefs were unimodal and symmetric, then subjects might find it "natural" to report the mean. Yet, this again requires using a strong assumption on an unobserved distribution.
\@startsection{section}{1} \z@{1.0\linespacing\@plus\linespacing}{.8\linespacing}{Mean vs Mode: Why the Belief Elicitation Scheme Matters}
\@startsection{subsection}{2} \z@{.8\linespacing\@plus.7\linespacing}{.7\linespacing}{Unobservability of Mean Beliefs in Theory-Driven Specifications}
To link the belief elicitation schemes above to actions/outcomes, consider an environment where an individual $i$ has a binary decision (e.g., to attend a protest or not, or whether to turn out to vote). This binary decision is denoted $P_i \in \{0,1\}$. Denote $P_{-i}$ as the proportion of other individuals choosing $P_i =1$, and the expected belief of $i$ over the latter as $\mathbb{E}_i P_{-i}$. For simplicity, let us write this best response as a linear specification, which is consistent with most of the empirical specifications cited above\footnote{It is clear that this linearity is only for exposition and to be closer to the linear and additive empirical specifications, as it can be derived from fundamentals of the game environment, such as linear-quadratic preferences.}:
The most common informational interventions cited in the previous sections manipulate the belief distribution over $P_{-i}$ to then manipulate $\mathbb{E}_i P_{-i}$ and identify $\delta_2$. For instance, bursztyn reveals the share of Saudi husbands who reported that their wives should work outside the home, while cyyz report the share of Hong Kong students who planned to attend a protest. The target parameter is $\delta_2$, which measures the extent to which individuals react to information about others' behavior. In particular, if $\delta_2>0$, it is a game of strategic complements, while if $\delta_2<0$, it is one of strategic substitutes.
The participation decision in equation (ref) is based on the mean of $i$'s belief distribution, which is consistent with expected-utility maximizing agents. However, if the elicitation task from (ref) is used, then subjects would report the mode of their belief distribution over $P_{-i}$, either exactly or approximately, and not the mean of their beliefs over $P_{-i}$. Thus, $\mathbb{E}_i P_{-i}$ is not generally observed in the subjects' responses to the researchers' questions. Hence, unless the belief distribution has an identical mean and mode, $\delta_2$ is not identified.
\@startsection{subsection}{2} \z@{.8\linespacing\@plus.7\linespacing}{.7\linespacing}{Means and Modes Can Be Updated in Opposite Directions}
Now, suppose that a researcher elicits modal beliefs through an elicitation scheme in Section (ref), but interprets it as though it was the mean belief with the goal of identifying $\delta_2$. It turns out that, if the mean and mode are actually not identical, then the researcher may not only fail to identify the magnitude of their desired effect, but may fail to identify the sign of the effect of interest.
To see this visually, we return to Figure (ref). The information intervention that reveals $x=0.17$ increases modal beliefs about $x$, while mean beliefs about $x$ decrease. Claim (ref) in Section (ref) formalizes sufficient conditions for this class of distributions upon which, if subjects are reporting modal beliefs, while the econometrician interpreted it as mean beliefs, the identified sign of $\delta_2$ in equation (ref) would be the opposite of the true value.
\@startsection{section}{1} \z@{1.0\linespacing\@plus\linespacing}{.8\linespacing}{Political Protests as Strategic Complements: An Empirical Application}
In this section, we revisit the empirical evidence in cyyz in light of the belief elicitation schemes discussed in Sections (ref) and (ref).
cyyz motivate their experimental design as being able to identify whether political protests are strategic complements (i.e., when a citizen believes more agents are likely to protest, she should become more likely to protest herself) or strategic substitutes (e.g., when citizens have an incentive to free-ride on the costly participation of others, as in tullock1971,palfrey1984,olsonlogic). This maps into whether $\delta_2$ in equation ((ref)) is positive or negative.
The strategic complementarity assumption has been widely used in recent models (e.g., mesquita,passarelli,barbera), leading gehlbach to write that strategic complementarity (SC, henceforth) “characterizes mass protest" (p.579).\footnote{It is also intuitive, as it can be rationalized if citizens enjoy an increased benefit from attending a larger protest through an increased likelihood of a regime change, or through increased social interactions from such behavior, or through a lower cost (one is less likely to be arrested amongst a larger crowd if there is a crackdown by the incumbent regime).} However, whether protests are strategic complements or substitutes has important welfare effects: political protests are strongly associated with regime changes (see chenoweth), but policy implications would differ whether protests are strategic substitutes or complements.
Within this context, cyyz conduct an important field experiment among students in Hong Kong. They find that potential protesters who were presented with the information that others were more likely to protest than their previously reported beliefs (elicited using the scheme in Section (ref)), became less likely to protest themselves. cyyz conclude that this is evidence against models that solely assume that protests are strategic complements. (p.1072)
We now show that cyyz's evidence can actually be completely consistent with SC, thereby providing an alternative interpretation of their results. Our arguments are based on: (i) that their belief elicitation scheme satisfies Claim (ref), thereby eliciting modal beliefs, (ii) cyyz still interpret reported beliefs as mean beliefs (using an analogue to equation (ref)), even though modes and means can be updated in opposite direction. Hence, they may identify an effect with opposite sign of their desired one.
We first overview their experiment, before illustrating these points using a stylized model and numerical exercises.
\@startsection{subsection}{2} \z@{.8\linespacing\@plus.7\linespacing}{.7\linespacing}{The Original Experiment}
The experiment in cyyz consists of three parts whose relevant features are overviewed here. Further details can be found in Appendix (ref).
In Part 1 of the experiment, one week before the planned protest, subjects were asked:
Question (i) elicits the ex-ante decision for each subject, and the authors find that 17% of subjects plan to attend the protest. Meanwhile, question (ii) elicits their (reported) prior belief about others' participation. As alluded to previously, it is a special case of the elicitation scheme in Claim (ref).
In Part 2 of the experiment (one day before the protest), all subjects were reminded of their Part 1 answers to (i) and (ii) and allowed to update the responses. However, a random subset of subjects (i.e, the “Treated" group) was given the information that 17% of subjects in the previous survey were planning to attend the protest. Part 3 of the experiment was conducted after the protest, and asked subjects to self-report whether they turned out ($P_i = 1$) or not ($P_i = 0$).
The authors estimate an analogue to equation (ref). Their target parameter is $\delta_2$, as they are in interested in how changes to one's beliefs affect participation decisions.
\@startsection{subsection}{2} \z@{.8\linespacing\@plus.7\linespacing}{.7\linespacing}{A Stylized Model: General Description}
In Appendix (ref), we provide a stylized model where we work through what an experimentalist would observe in terms of elicited beliefs and participation decisions, if subjects maximized their expected utility in cyyz's set-up. Here, we give a general outline of the model's set-up and insights.
Subjects in the model make two choices. They attend the protest ($P_i = 1$), or not ($P_i = 0$), and they choose some value to report, $r_i$, about others' participation, given their beliefs and the researchers' incentives described in Section (ref). As a rhetorical device, we construct subjects' preferences to satisfy SC: i.e., their preferences to attend are increasing in others' participation levels, $P_{-i}$. There is a cost of $0.165$ to attend the protest, so that $i$ wants to attend if the mean of others' attendance is larger than $16.5\%$. Critically, given the incentives used for belief-elicitation, even though the actual decision to protest depends on mean beliefs, the decision on what to report depends on modal beliefs.
Subjects can belong to two groups, which only differ in their distribution of beliefs over $P_{-i}$. Both groups have prior mean beliefs below $0.165$, so neither wants to attend the protest ex-ante. However, Group 1 has a high modal belief over $P_{-i}$, while Group 2 has low modal beliefs. Following the global games literature (see morris for an overview), we model the information intervention (Part 2 of the experiment) as a noisy signal which has a distribution centered around the true value (17%, based on the information intervention in cyyz). This information makes subjects update their beliefs, which yields a posterior distribution from which we calculate posterior means and modes over others' attendance. Prior and posterior means and modes in this stylized environment are summarized in Table (ref).
Suppose that subjects reveal their true intentions when asked about their willingness to protest. Then, following the intervention, only Group 1 subjects would express a willingness to participate in the protest following treatment (as their posterior mean belief is above the cost, 0.165). However, the elicitation scheme in Section (ref) satisfies Claim (ref), so subjects should report their modal beliefs. This leads to the following two results (Claims (ref) and (ref)) that would seem to violate SC if one interpreted “elicited beliefs" as mean beliefs.
Suppose that a part of Group 1 was randomly assigned to a control group (who did not receive information intervention), and the rest of Group 1 to a treatment group (who received the information intervention).
By comparison, this misinterpretation would not occur if one used the elicitation scheme in Claim (ref), as both groups would have reported mean beliefs, which increase following treatment. However, if no incentives were provided to subjects (as in Section (ref)), then we cannot derive best responses for $r_i$ in this example.
\@startsection{subsection}{2} \z@{.8\linespacing\@plus.7\linespacing}{.7\linespacing}{Means vs. Modes in cyyz: A Quantification}
While we cannot truly know whether beliefs were asymmetric and whether means and modes were updated in opposite directions (and even whether modes themselves were reported), we can illustrate that this concern may be quantitatively meaningful. To do so, we extend the previous example to a more easily quantifiable set-up, where we can provide sufficient conditions under which the mean and mode beliefs would be updated in opposite directions.
Let the true proportion of students planning to attend the protest be $x \in [0,1]$. Assume that each subject, $i$, has beliefs over $x$ that follow a $Beta(\alpha_i,\beta_i)$ distribution, where $\alpha_i, \beta_i$ are parameters larger than 1.\footnote{The $Beta$ distribution has several advantages in this set-up. First, its support is bounded between $[0,1]$, just like the share of participation turnout. Second, it can be symmetric ($\alpha_i=\beta_i$) with mean and mode being equal, or asymmetric, allowing us to test our mechanism which relies on asymmetric beliefs. Third, its mode, mean and variances have closed form solutions. Fourth, for a $Beta$ distribution, condition (ii) from Claim (ref) holds, and hence subjects should be reporting values approximately equal to the prior and posterior modes. Finally, it provides the convenience of being the conjugate prior when the likelihood is binomial/Bernoulli, which would be useful to model how subjects update their priors after the particular information intervention used in cyyz. The latter is used explicitly in our proof of Claim (ref).}
The information intervention in cyyz exposes subjects to the true level of planned participation of 1234 subjects, and in the experiment $\hat x=17\%$ of the 1234 subjects planned to protest. We assume that subjects interpret this information as a Binomial draw from the true, but unobserved, realized proportion $x^*$ with sample size $n = 1234$. Then:
Claim (ref) reinforces Claims (ref) and (ref), while generalizing Figure (ref): means and modes are often updated in opposite directions, and thus misinterpreting modes as means could reverse the relationship between participation choices and the direction of belief-update.
We can use this set-up in a numerical exercise. Let us interpret the belief data from cyyz as modal beliefs from a heterogenous $Beta$ distribution. As we observe different reports across individuals (which are the modes of their corresponding belief distributions), such beliefs must be heterogeneous across participants.\footnote{The only alternative would be to attribute these differences to measurement error, which would be unsatisfactory. The parametric assumptions then allow us to learn about the distributions of individual beliefs.}
We model this heterogeneity by assuming a “random coefficients" type structure on the parameters $\alpha_i, \beta_i$, yielding a Bayesian hierarchical model. In particular, to be consistent with the structure above, we assume $\alpha_i, \beta_i$ are i.i.d. across the population according to: $\alpha_i - 1 \sim \chi^2(\ell)$ and $\beta_i-1 \sim \chi^2(q)$, with the two distributions being independent. Next, the properties of $\chi^2$-distributions imply that the modal beliefs elicited in their experiment follow:
Hence, the researcher with the dataset from cyyz observes the distribution of modes across subjects, characterized by ((ref)), rather than individual belief distributions ($Beta(\alpha_i,\beta_i)$). Note that the former can be symmetric ($\ell=q$) even when individual beliefs are not ($\alpha_i \neq \beta_i$).\footnote{In footnote 25, cyyz suggest mean and mode are likely to be close together. However, if they only have data on reported modes, they can only compare the mean and modes of the distribution of reported modes, rather than means and modes of the original distribution of individual beliefs.}
Identification of $\ell,q$ in ((ref)) follows immediately from the moments of the distribution of modes in the data. We estimate $\ell/2, q/2$ by Maximum Likelihood, yielding estimates $\hat{\ell}/2, \hat{q}/2$. Since $\ell,q$ are degrees of freedom, our estimates for them are given by their rounded up counterparts. The estimates are shown in Appendix Table (ref). Appendix Table (ref) shows that this parametrization fits the moments in the data well.
To calculate the share of beliefs with modes and means being updated in opposite directions (i.e. that satisfy the conditions outlined in Claim (ref), in which those two statistics are on opposite sides of 0.17), we draw $R=100000$ parameters from the joint estimated distribution of $\{(\alpha_i, \beta_i)\}_{i=1}^{n}$. We then compute the share of such draws that have the mode and mean on opposite sides of 17%, and such that the mode $\pm 2$ percentage points has more mass than the mean $\pm 2$ percentage points. This check is needed given the authors' error tolerance.
The main result from this exercise is that, even in a simplified set-up, $34.8\%$ of drawn beliefs would have mean and modes updated in opposite directions in the original experiment where subjects would have preferred to report the mode rather than the mean. While this is merely illustrative, it does suggest that this may be a meaningful concern, and that the authors' original estimates could have had their signs reversed if they interpreted “elicited beliefs" as modal beliefs.
\@startsection{section}{1} \z@{1.0\linespacing\@plus\linespacing}{.8\linespacing}{Concluding Remarks}
There has been increased popularity in using experiments to measure and vary individual beliefs, particularly in strategic settings. We discuss how belief elicitation schemes that look prima-facie similar, and that are widely used in experimental work, may identify different properties of the true probability distribution. In particular, we describe cases where incentive schemes rewarding subjects for guessing an outcome correctly incentivize reports of modal beliefs, mean beliefs or median beliefs. We also discuss the perils of not using an incentive scheme at all. While it is perfectly possible that, in certain settings, belief distributions are indeed symmetric and unimodal, in others they may not, which could imply that even the sign of identified effects may be switched if one interprets mean beliefs as modal beliefs and vice-versa. Hence, researchers should consider which scheme would elicit the appropriate data for their research question.
We illustrate these points using the experimental data in cyyz within the context of political protests in Hong Kong. The authors provide an innovative way of examining how the decision to participate changes when there is an exogenous shock to citizen's beliefs. We make two main observations. First, our results imply that unless one assumes symmetric unimodal distributions, the elicitation method in cyyz measures modal beliefs instead of the mean beliefs that the regression specification calls for. Second, we explain how interpreting modal beliefs as mean beliefs might lead researchers to erroneously reject strategic complementarity. This suggests alternative interpretations to their findings which could be studied in future work.