Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
85,388 characters · 8 sections · 0 citation commands
Cheating with (Recursive) Models
Agents in economic models rely on mental models of their environment for quantifying correlations between variables, the most important of which describes the payoff consequences of their own actions. The vast majority of economic models assume rational expectations - i.e., the agent's subjective models coincide with the modeler's, such that the agent's belief is consistent with the true data-generating process. Yet when the agent's subjective model is misspecified, his predicted correlations may deviate from the truth. This difficulty is faced not only by agents in economic models but also by real-life researchers in physical and social sciences, who make use of statistical models. This paper poses a simple question: How far off can the correlations predicted by misspecified models get?
To make our question concrete, imagine an analyst\ who wishes to demonstrate to an audience that two variables, $x$ and $y$, are strongly related. Direct evidence about the correlation between these variables is hard to come by. However, the analyst has access to data about the correlation of $x$ and $y$ with other variables. He therefore constructs a $model$ that involves $x$, $ y $ and a selection of auxiliary variables. He fits this model to a large sample and uses the estimated model to predict the correlation between $x$ and $y$. The analyst is unable (or unwilling) to tamper with the data. However, he is free to choose the auxiliary variables and how they operate in the model. To what extent does this degree of freedom enable the analyst to attain his underlying objective?
This hypothetical scenario is inspired by a number of real-life situations. First, academic researchers often serve as consultants to policy makers or activist groups in pursuit of a particular agenda.\ E.g., consider an economist consulting a policy maker who pursues a tax-cutting agenda and seeks intellectual support for this position. The policy maker would therefore benefit from an academic study showing a strong quantitative relation between tax cuts and economic growth. Second, the analyst may be wedded to a particular stand regarding the relation between $x$ and $y$ because he staked his public reputation on this claim in the past. Finally, the analyst may want to make a splash with a counterintuitive finding and will stop exploring alternative model specifications once he obtains such a result.
We restrict the class of models that the analyst can employ to be recursive linear-regression models. A model in this familiar class consists of a list of linear-regression equations, such that an explanatory variable in one equation cannot appear as a dependent variable in another equation down the list. We assume that the recursive model includes the variables $x$ and $y$, as well as a selection of up to $n-2$ additional variables. Thus, the total number of variables in the analyst's model is $n$, which is a natural measure of the model's complexity. Each equation is estimated via Ordinary Least Squares (OLS) against an arbitrarily large (and unbiased) sample.
The following quote lucidly summarizes two attractions of recursive models:
The causal interpretation of recursive models is particularly resonant. If $x$ appears exclusively as an explanatory variable in the system of equations while $y$ appears exclusively as a dependent variable, the recursive model intuitively charts a causal explanation that pits $x$ as a primary cause of $y$, such that the estimated correlation between $x$ and $ y$ can be legitimately interpreted as an estimated $causal$ effect of $x$ on $y$.
A three-variable example
To illustrate our exercise, suppose that the analyst estimates the following three-variable recursive model:
where $x_{1},x_{2},x_{3}$ all have zero mean and unit variance. The analyst assumes that all the $\Greekmath 0122 _{k}$'s are mutually uncorrelated, and also that for every $k>1$ and $j<k$, $\Greekmath 0122 _{k}$ is uncorrelated with $x_{j}$ (for $j=k-1$, this is mechanically implied by the OLS method).
For a real-life situation behind this example, consider a pharmaceutical company that introduces a new drug, and therefore has a vested interest in demonstrating a large correlation between the dosage of its active ingredient ($x_{1}$) and the ten-year survival rate associated with some disease. This correlation cannot be directly measured in the short run. However, past experience reveals the correlations between the ten-year survival rate and the levels of various bio-markers (which can serve as the intermediate variable $x_{2}$). The correlation between these markers and the drug dosage can be measured experimentally in the short run. Thus, on one hand the situation calls for a model in the manner of ((ref)), yet on the other hand the pharmaceutical company's R&D unit may select the bio-marker $x_{2}$ opportunistically, in order to get a large estimated effect. Of course, in reality there may be various checks and balances that will constrain this opportunism. However, it is interesting to know how badly it can get, in order to evaluate the importance for these checks and balances.
Let $\Greekmath 011A _{ij}$ denote the correlation between $x_{i}$ and $x_{j}$ according to the $true$ data-generating process. Suppose that $x_{1}$ and $ x_{3}$ are objectively uncorrelated - i.e. $r=\Greekmath 011A _{13}=0$. The estimated correlation between these variables according to the model, given the analyst's procedure and its underlying assumptions, is
It is easy to see from this expression how the model can generate spurious estimated correlation between $x_{1}$ and $x_{3}$, even though none exists in reality. All the analyst has to do is select a variable $x_{2}$ that is positively correlated with both $x_{1}$ and $x_{3}$, such that $\Greekmath 011A _{12}\cdot \Greekmath 011A _{23}>0$.
But how large can the false estimated correlation $\hat{\Greekmath 011A }_{13}$ be? Intuitively, since $x_{1}$ and $x_{3}$ are objectively uncorrelated, if we choose $x_{2}$ such that it is highly correlated with $x_{1}$, its correlation with $x_{3}$ will be low. In other words, increasing $\Greekmath 011A _{12}$ will come at the expense of decreasing $\Greekmath 011A _{23}$. Formally, consider the true correlation matrix:
By definition, this matrix is positive semi-definite. This property is characterized by the inequality $(\Greekmath 011A _{12})^{2}+(\Greekmath 011A _{23})^{2}\leq 1$. The maximal value of $\Greekmath 011A _{12}\cdot \Greekmath 011A _{23}$ subject to this constraint is $\frac{1}{2}$, and therefore this is the maximal false correlation that the above recursive model can generate. This bound is tight: It can be attained if we define $x_{2}$ to be a deterministic function of $x_{1}$ and $ x_{3}$, given by $x_{2}=\frac{1}{2}(x_{1}+x_{3})$. Thus, while a given misspecified recursive model may be able to generate spurious estimated correlation between objectively independent variables, there is a limit to how far it can go.
Our interest in the upper bound on $\hat{\Greekmath 011A }_{13}$ is not purely mathematical. As the above \textquotedblleft bio-marker\textquotedblright\ example implies, we have in mind situations in which the analyst can select $ x_{2}$ from a large pool of potential auxiliary variable. In the current age of \textquotedblleft big data\textquotedblright , analysts have access to datasets involving a huge number of covariates. As a result, they have considerable freedom when deciding which variables to incorporate into their models. This helps our analyst generate a false correlation that approaches the theoretical upper bound.
To give a concrete demonstration for this claim, consider Figure $1$, which is extracted from a database compiled by the World Health Organization and collected by Reshef et al. (2011).\footnote{ The variables are collected on all countries in the WHO database (see www.who.int/whosis/en/) for the year 2009.} All the variables are taken from this database. The figure displays the maximal $\hat{\Greekmath 011A }_{13}$ correlation that the model ((ref)) can generate for two fixed pairs of variables $x_{1}$ and $x_{3}$ with $\Greekmath 011A _{13}$ close to zero, when the auxiliary variable is selected from a pool whose size is given by the horizontal axis (variables were added to the pool in some arbitrary order). When the analyst can choose $x_{2}$ from only ten possible auxiliary variables, the estimated correlation between $x_{1}$ and $x_{3}$ he can generate with ((ref)) is still modest. In contrast, once the pool size is in the hundreds, the estimated correlation approaches the upper bound of $\frac{1}{2}$.
For a specific variable that gets us near the theoretical upper bound, consider the figure's R.H.S, where $x_{1}$ represents urban population and $ x_{3}$ represents liver cancer deaths per 100,000 men. The true correlation between these variables is $0.05$. If the analyst selects $x_{2}$ to be coal consumption (measured in tonnes oil equivalent), the estimated correlation between $x_{1}$ and $x_{3}$ is $0.43$, far above the objective value. This selection of $x_{2}$ has the added advantage that the model suggests an intuitive causal mechanism: Urbanization causes cancer deaths via its effect on coal consumption.
Review of the results
We present our formal model in Section 2 and pose our main problem: What is the largest estimated correlation between $x_{1}$ and $ x_{n} $ that a recursive, $n$-variable linear-regression model can generate? We impose one constraint on this maximization problem: While the estimated model is allowed to distort correlations among variables, it must produce correct estimates of the individual variables' mean and variance. The linearity of the analyst's model implies that only the latter has bite.
To motivate this constraint, recall that the scenario behind our model involves a sophisticated analyst and a lay audience. Because the audience is relatively unsophisticated, it cannot be expected to discipline the analyst's opportunistic model selection with elaborate tests for model misspecification that involve conditional or unconditional correlations. However, monitoring individual variables is a much simpler task than monitoring correlations between variables. E.g., it is relatively easy to disqualify an economic model that predicts highly volatile inflation if observed inflation is relatively stable. Likewise, a climatological model that underpredicts temperature volatility loses credibility, even for a lay audience.
Beyond this justification, we simply find it intrinsically interesting to know the extent to which misspecified recursive models can distort correlations between variables while preserving moments of individual variables. At any rate, we relax the constraint in Section 4, for the special case of models that consist of a single non-degenerate regression equation. We use this case to shed light on the analyst's opportunistic use of \textquotedblleft bad controls\textquotedblright .
In Section 3, we derive the following result. For a generic objective covariance matrix with $\Greekmath 011A _{1n}=r$, the maximal estimated correlation $ \hat{\Greekmath 011A }_{1n}$ that a recursive model with up to $n$ variables can generate, subject to preserving the mean and variance of individual variables, is
The upper bound given by ((ref)) is tight. Specifically, it is attained by the simplest recursive model that involves $n$ variables: For every $k=2,...,n$, $x_{k}$ is regressed on $x_{k-1}$ only. This model is represented graphically by the chain $x_{1}\rightarrow x_{2}\rightarrow \cdots \rightarrow x_{n}$. his chain has an intuitive causal interpretation, which enables the analyst to present $\hat{\Greekmath 011A }_{1n}$ as an estimated causal effect of $x_{1}$ on $x_{n}$. The variables $x_{2},...,x_{n-1}$ that are employed in this model have a simple definition, too: They are all deterministic linear functions of $x_{1}$ and $x_{n}$.
Formula ((ref)) reproduces the value $\hat{\Greekmath 011A }_{13}=\frac{1}{2}$ that we derived in our illustrative example, and it is strictly increasing in $n$. When $n\rightarrow \infty $, the expression converges to $1$. That is, regardless of the true correlation between $x_{1}$ and $x_{n}$, a sufficiently large recursive model can generate an arbitrarily large estimated correlation. The lesson is that when the analyst is free to select his model and the variables that inhabit it, he can deliver any conclusion about the effect of one variable on another - unless we impose constraints on his procedure, such as bounds on the complexity of his model or additional misspecification tests.
The formula ((ref)) has a simple geometric interpretation, which also betrays the construction of the recursive model and objective covariance matrix that implement the upper bound. Take the angle that represents the objective correlation $r$ between $x_{1}$ and $x_{n}$; divide it into $n-1$ equal sub-angles; this sub-angle represents the correlation between adjacent variables along the above causal chain; the product of these correlations produces the estimated correlation between $x_{1}$ and $ x_{n}$.
The detailed proof of our main result is presented in\ Section 5. It relies on the graphical representation of recursive models and employs tools from the Bayesian-networks literature (Cowell et al. (1999), Koller and Friedman (2009)). In the Appendix, we present partial analysis of our question for a different class of models involving $binary$ variables.
Related literature
There is a huge literature on misspecified models in various branches of economics and statistics, which is too vast to survey in detail here. A few recent references can serve as entry points for the interested reader: Esponda and Pouzo (2016), Bonhomme and Weidner (2018) and Molavi (2019). To our knowledge, our paper is the first to carry out a worst-case analysis of misspecified models' predicted correlations.
The \textquotedblleft analyst\textquotedblright\ story that motivated this exercise brings to mind the phenomenon of researcher bias. A few works in Economics have explicitly modeled this bias and its implications for statistical inference. (Of course, there is a larger literature on how econometricians should cope with researcher/publication bias, but here we only describe exercises that contain explicit models of the researcher's behavior.) Leamer (1974) suggests a method of discounting evidence when linear regression models are constructed after some data have been partially analyzed. Lovell (1983) considers a researcher who chooses $k$ out of $n$ independent variables as explanatory variables in a single regression with the aim of maximizing the coefficient of correlation between the chosen variables and the dependent variable. He argues that a regression coefficient that appears to be significant at the $\Greekmath 010B $ level should be regarded as significant at only the $1-(1-\Greekmath 010B )^{n/k}$ level. Glaeser (2008) suggests a way of correcting for this form of data mining in the coefficient estimate.
More recently, Di-Tillio, Ottaviani and Sorensen (2017,2019) characterize data distributions for which strategic sample selection (e.g., selecting the $k$ highest observations out of $n$) benefits an evaluator who must take an action after observing the selected sample realizations. Finally, Spiess (2018) proposes a mechanism-design framework to align the preferences of the researcher with that of \textquotedblleft society\textquotedblright :\ A social planner first chooses a menu of possible\ estimators, the investigator chooses an estimator from this set, and the estimator is then applied to the sampled observations.
The analyst in our model\ need not be a scientist - he could also be a politician or a pundit. Under this interpretation, constructing a model and fitting it to data is not an explicit, formal affair. Rather, it involves spinning a \textquotedblleft narrative\textquotedblright\ about the effect of policy on consequences and using casual empirical evidence to substantiate it. Eliaz and Spiegler (2018) propose a model of political beliefs that is based on this idea. From this point of view, our exercise in this paper explores the extent to which false narratives can exaggerate the effect of policy.
Let $p$ be an objective probability measure over $n$ variables, $ x_{1},...,x_{n}$. For every $A\subset \{1,...,n\}$, denote $ x_{A}=(x_{i})_{i\in A}$. Assume that the marginal of $p$ on each of these variables has\ zero mean and unit variance. This will entail no loss of generality for our purposes. We use $\Greekmath 011A _{ij}$ to denote the coefficient of correlation between the variables $x_{i},x_{j}$, according to $p$. In particular, denote $\Greekmath 011A _{1n}=r$. The covariance matrix that characterizes $p$ is therefore $(\Greekmath 011A _{ij})$.
An analyst estimates a recursive model that involves these variables. This model consists of a system of linear-regression equations. For every $ k=1,...,n$, the $k^{th}$ equation takes the form
where:
We refer to such a system of regression equations as an $n$-variable recursive model. The function $R$ effectively defines a directed acyclic graph (DAG) over the set of nodes $\{1,...,n\}$, such that a link $ i\rightarrow j$ exists whenever $i\in R(j)$. DAGs are often interpreted as causal models (see Pearl (2009)). We will make use of the DAG representation in the proof of our main result, as well as in the Appendix. We will also allude to the recursive model's causal interpretation, but without addressing explicitly the question of causal inference. The analyst in our model engages in a problem of fitting a model to data. While the causal interpretation may help the analyst to \textquotedblleft sell\textquotedblright\ the model to his audience, he does not engage in an explicit procedure for drawing causal inference from his model.
Note that in the analyst's model, the equation for $x_{1}$ has no explanatory variables, and $x_{n}$ is not an explanatory variable in any equation. Furthermore, the partial ordering given by $R$ is consistent with the natural enumeration of variables (i.e., $i\in R(j)$ implies $i<j$). This restriction is made for notational convenience; relaxing it would not change our results. However, it has the additional advantage that the causal interpretation of the model-estimated correlation between $x_{1}$ and $x_{n}$ is sensible. Indeed, it is legitimate according to Pearl's (2009) rules for causal inference based on DAG-represented models.
The analyst's assumption that each $\Greekmath 0122 _{k}$ is uncorrelated with $ x_{R(k)}$ is redundant, because it is an automatic consequence of his OLS procedure for estimating $\Greekmath 010C _{k}$. In contrast, his assumption that $ \Greekmath 0122 _{k}$ is uncorrelated with all other $x_{j}$, $j<k$, is the basis for how he combines the individually estimated equations into a joint estimated distribution. It is fundamentally a conditional-independence assumption - namely, that $x_{k}$ is independent of $(x_{j})_{j\in \{1,..,k-1\}-R(k)}$ conditional on $x_{R(k)}$. This assumption will typically be false - indeed, it is what makes his model misspecified and what enables him to \textquotedblleft cheat\textquotedblright\ with his model.
Under this assumption, the analyst proceeds to estimate the correlation between $x_{1}$ and $x_{n}$, which can be computed according to the following recursive procedure. Start with the $n^{th}$ equation. For every $ j\in R(n)-\{1\}$, replace $x_{j}$ with the R.H.S of the $j^{th}$ equation. This produces a new equation for $x_{n}$, with a different set of explanatory variables. Repeat the substitution for each one of these variables, and continue doing so until the only remaining explanatory variable is $x_{1}$.
The procedure's final output is the thus the equation
The coefficients $\Greekmath 010B ,\Greekmath 010D _{1},...,\Greekmath 010D _{n}$ are combinations of $ \Greekmath 010C $ parameters (which were obtained by OLS estimation of the individual equations). Likewise, the distribution of each error term $\Greekmath 0122 _{j}$ is taken from the estimated $j^{th}$ equation.
The analyst uses this equation to estimate the variance of $x_{n}$ and its covariance with $x_{1}$, implementing the (partly erroneous) assumptions that all the $x_{k}$'s have zero mean and unit variance, and that the $ \Greekmath 0122 _{k}$'s mean, mutual covariance and covariance with $x_{1}$ are all zero:
and
Therefore, the estimated coefficient of correlation between $x_{1}$ and $ x_{n}$ is
We assume that the analyst faces the constraint that the estimated mean and variance of all individual variables must be correct (see the Introduction for a discussion of this constraint). The requirement that the estimated means of individual variables are undistorted has no bite: The OLS procedure for individual equations satisfies it. Thus, the constraint is reduced to the requirement that
for all $k$ (we can calculate this estimated variance for every $k>1$, using the same recursive procedure we applied to $x_{n}$). This reduces ((ref)) to
Our objective will be to examine how large this expression can be, given $n$ and a generic objective covariance matrix problem.
Comments on the analyst's procedure
In our model, the analyst relies on a structural model to generate an estimate of the correlation between $x_{1}$ and $x_{n}$, which he presents to a lay audience. The process by which he selects\ the model remains hidden from the audience. But why does the analyst use a model to estimate the correlation between $x_{1}$ and $x_{n}$, rather than estimating it $directly$? One answer may be that direct evidence on this correlation is hard to come by (as, for example, in the case of long-term health effects of nutritional choices). In this case, the analyst $must$ use a model to extrapolate an estimate of $\Greekmath 011A _{1n}$ from observed data.
Another answer is that analysts use models as simplified representations of a complex reality, which they can consult for $multiple$ conditional-estimation tasks: Estimating the effect of $x_{1}$ on $x_{n}$ is only one of these tasks. This is illustrated by the following quote: \textquotedblleft The economy is an extremely complicated mechanism, and every macroeconomic model is a vast simplification of reality\ldots the large scale of FRB/US [a general equilibrium model employed by the Federal Reserve Bank - the authors] is an advantage in that it can perform a wide variety of computational `what if' experiments.\textquotedblright \footnote{ This quote is taken from a speech by Stanley Fisher: See https://www.federalreserve.gov/newsevents/speech/fischer20170211a.htm.} From this point of view, our analysis concerns the maximal distortion of pairwise correlations that such models can produce.
Our treatment of the model accommodates both interpretations. In particular, we will allow for the possibility that $1\in R(n)$, which means that the analyst does have data about the joint distribution of $x_{1}$ and $x_{n}$. As we will see, our main result will not make use of this possibility.
For every $r,n$, denote
We are now able to state our main result.
Let us illustrate the upper bound given by Theorem (ref) numerically for the case of $r=0$, as a function of $n$:
As we can see, the marginal contribution of adding a variable to the false correlation that the analyst's model can produce decays quickly. However, when $n\rightarrow \infty $, the upper bound converges to one. This is the case for any value of $r$. That is, even if the true correlation between $ x_{1}$ and $x_{n}$ is strongly negative, a sufficiently large model can produce a large positive correlation.
The recursive model that attains the upper bound has a simple structure. Its DAG representation is a single chain
Intuitively, this is the simplest connected DAG with $n$ nodes: It has the smallest number of links among this class of DAGs, and it has no junctions. The distribution over the auxiliary variables $x_{2},...,x_{n}$ in the upper bound's implementation has a simple structure, too: Every $x_{k}$ is a different linear combination of two independent \textquotedblleft factors\textquotedblright , $s_{1}$ and $s_{2}$. We can identify $s_{1}$ with $x_{1}$, without loss of generality. The closer the variable lies to $ x_{1}$ along the chain, the larger the weight it puts on $s_{1}$.$\medskip $
General outline of the proof
The proof of Theorem (ref) proceeds in three major steps. First, the constraint that the estimated model preserves the variance of individual variables for a generic objective distribution reduces the class of candidate recursive models to those that can be represented by perfect DAGs. Since perfect DAGs preserve marginals of individual variables for $every$ objective distribution (see Spiegler (2017)), the theorem can be stated more strongly for this subclass of recursive models.
That is, when the recursive model is represented by a perfect DAG, the upper bound on $\hat{\Greekmath 011A }_{1n}$ holds for $any$ objective covariance matrix, and the undistorted-variance constraint is redundant.
In the second step, we use the tool of junction trees in the Bayesian-networks literature (Cowell et al. (1999)) to perform a further reduction in the class of relevant recursive models. Consider a recursive model represented by a non-chain perfect DAG. We show that the analyst can generate the same $\hat{\Greekmath 011A }_{1n}$ with another objective distribution and a recursive model that takes the form of a simple chain $1\rightarrow \cdots \rightarrow n$. Furthermore, this chain will involve no more variables than the original model.
To illustrate this argument, consider the following recursive model with $ n=4 $:
This recursive model has the following DAG representation:
Because $x_{4}$ depends on $x_{2}$ and $x_{3}$ only through their linear combination $\Greekmath 010C _{24}x_{2}+\Greekmath 010C _{34}x_{3}$, we can replace $ (x_{2},x_{3})$ with a scalar variable $x_{5}$, such that the recursive model becomes
This model is represented by the DAG $1\rightarrow 5\rightarrow 3$, which is a simple chain that consists of fewer nodes than the original DAG.
This means that in order to calculate the upper bound on $\hat{\Greekmath 011A }_{1n}$, we can restrict attention to the chain model. But in this case, the analyst's objective function has a simple explicit form:
Thus, in the third step, we derive the upper bound by finding the correlation matrix that maximizes the R.H.S of this formula, subject to the constraints that $\Greekmath 011A _{1n}=r$ and that the matrix is positive semi-definite (which is the property that defines the class of covariance matrices). The solution to this problem has a simple geometric interpretation.
Analysts often propose models that take the form of a $single$ linear-regression equation, consisting of a dependent variable $x_{n}$ (where $n>2$), an explanatory variable $x_{1}$ and $n-2$ \textquotedblleft $ control$\textquotedblright\ variables $x_{2},...,x_{n-1}$ . Using the language of Section 2, this corresponds to the specification $R(k)=\emptyset $ for all $k=1,...,n-1$ and $R(n)=\{1,...,n-1\}$. That is, the only non-degenerate equation is the one for $x_{n}$, hence the term \textquotedblleft single-equation model\textquotedblright . Note that in this case, the OLS regression coefficient $\Greekmath 010C _{1n}$ in the equation for $ x_{n}$ coincides with $\hat{\Greekmath 011A }_{1n}=\Greekmath 010B /\widehat{Var}(x_{n})$, as defined in Section 2.
Using the graphical representation, the single-equation model corresponds to a DAG in which $x_{1},...,x_{n-1}$ are all ancestral nodes that send links into $x_{n}$. Since this DAG is imperfect, Lemma (ref) in Section 5 implies that for almost all objective covariance matrices, the estimated variance of $x_{n}$ according to the single-equation model will differ from its true value.\footnote{ All the other variables are represented by ancestral nodes, and therefore their marginals are not distorted (see Spiegler (2017)).} However, given the particular interest in this class of models, we relax the correct-variance constraint in this section and look for the maximal false correlation that such models can generate. For expositional convenience, we focus on the case of $r=0$.
Thus, to magnify the false correlation between $x_{1}$ and $x_{n}$, the analyst would select the \textquotedblleft control\textquotedblright\ variables such that a certain linear combination of them has strong negative correlation with $x_{1}$. That is, the analyst will prefer his regression model to exhibit multicollinearity. This inflates the estimated variance of $ x_{n}$; indeed, $\widehat{Var}(x_{3})\rightarrow \infty $ when $\Greekmath 010E \rightarrow -1$. However, at the same time it increases the estimated covariance between $x_{1}$ and $x_{3}$, which more than compensates for this increase in variance. As a result, the estimated correlation between $x_{1}$ and $x_{3}$ rises substantially.
The three-variable model that implements the upper bound is represented by the DAG $1\rightarrow 3\leftarrow 2$. That is, it treats the variables $ x_{1} $ and $x_{2}$ as independent, even though in reality they are correlated. In particular, the objective distribution may be consistent with a DAG that adds a link $1\rightarrow 2$ to the analyst's DAG, such that adding $x_{2}$ to the regression means that we control for a \textquotedblleft post-treatment\textquotedblright\ variable (where $x_{1}$ is viewed as the treatment). In other words, $x_{2}$ is a \textquotedblleft bad control\textquotedblright\ (see Angrist and Pischke (2008), p. 64). \footnote{ See also http://causality.cs.ucla.edu/blog/index.php/2019/08/14/a-crash-course-in-good-and-bad-control/. }
The upper bound of $1/\sqrt{2}$ in Proposition (ref) is obtained with $ n=3$. Recall that under the undistorted-variance constraint, the upper bound on $\hat{\Greekmath 011A }_{1,n}$ for $n=3$ is $1/2$. This shows that the constraint has bite. However, when $n$ is sufficiently large, the single-equation model is outperformed by the multi-equation chain model, which does satisfy the undistorted-variance constraint.
The proof relies on concepts and tools from the Bayesian-network literature (Cowell et al. (1999), Koller and Friedman (2009)). Therefore, we introduce a few definitions that will serve us in the proof.
A DAG is a pair $G=(N,R)$, where $N$ is a set of nodes and $R\subset N\times N$ is a pair of directed links. We assume throughout that $N=\{1,...,n\}$. With some abuse of notation, $R(i)$ is the set of nodes $j$ for which the DAG includes a link $j\rightarrow i$. A DAG is $perfect$ if whenever $i,j\in R(k)$ for some $i,j,k\in N$, it is the case that $i\in R(j)$ or $j\in R(i)$.
A subset of nodes $C\subseteq N$ is a $clique$ if for every $i,j\in C$, $iRj$ or $jRi$. We say that a clique is maximal if it is not contained in another clique. We use $\mathcal{C}$ to denote the collection of maximal cliques in a DAG.
A node $i\in N$ is ancestral if $R(i)$ is empty. A node $i\in N$ is terminal if there is no $j\in N$ such that $i\in R(j)$. In line with our definition of recursive models in Section 2, we assume that $1$ is ancestral and $n$ is terminal. It is also easy to verify that we can restrict attention to DAGs in which $n$ is the $only$ terminal node - otherwise, we can remove the other terminal nodes from the DAG, without changing $\hat{p}(x_{n}\mid x_{1}) $. We will take these restrictions for granted henceforth.
The analyst's procedure for estimating $\hat{\Greekmath 011A }_{1n}$, as described in Section 2, has an equivalent description in the language of Bayesian networks, which we now describe.
Because the analyst estimates a linear model, it is as if he believes that the underlying distribution $p$ is multivariate normal , where the estimated $k^{th}$ equation is a complete description of the conditional distribution $(p(x_{k}\mid x_{R(k)}))$. Therefore, from now, we will proceed as if $p$ were indeed a standardized multivariate normal with covariance matrix $(\Greekmath 011A _{ij})$, such that the $k^{th}$ regression equation corresponds to measuring the correct distribution of $x_{k}$ conditional on $ x_{R(k)}$. This is helpful expositionally and entails no loss of generality.
Given an objective distribution $p$ over $x_{1},...,x_{n}$ and a DAG $G$, define the Bayesian-network factorization formula:
We say that $p$ is consistent with $G$ if $p_{G}=p$.
By Koller and Friedman (2009, Ch. 7), when $p$ is multivariate normal, $ p_{G} $ is reduced to the estimated joint distribution as described in Section 2. In particular, we can use $p_{G}$ to calculate the estimated marginal of $x_{k}$ for any $k$:
Likewise, the induced estimated distribution of $x_{n}$ conditional on $ x_{1} $ is
This conditional distribution, together with the marginals $p_{G}(x_{1})$ and $p_{G}(x_{n})$, induce the estimated correlation coefficient $\hat{\Greekmath 011A } _{1n}$ given by ((ref)).
Because we take $p$ to be multivariate normal, the constraint that $p_{G}$ does not distort the mean and variance of individual variables is equivalent to the requirement that the estimated marginal distribution $(p_{G}(x_{k}))$ coincides with the objective marginal distribution of $x_{k}$. This constraint necessarily holds if $G$ is perfect. Furthermore, when $G$ is perfect, $p_{G}(x_{C})\equiv p(x_{C})$ for every clique $C$ in $G$ (see Spiegler (2017)).
Our first step is to establish that for generic $p$, perfection is necessary for the correct-marginal constraint.
The next step is based on the following definition.
A linear DAG is thus a causal chain $1\rightarrow \cdots \rightarrow n$. Every linear DAG is perfect by definition.
To solve the reduced problem we have arrived at, we first note that
Thus, the problem of maximizing $\hat{\Greekmath 011A }_{1n}$ is equivalent to maximizing the product of terms in a symmetric $n\times n$ matrix, subject to the constraint that the matrix is positive semi-definite, all diagonal elements are equal to one, and the $(1,n)$ entry is equal to $r$:
Note that the positive semi-definiteness constraint is what makes the problem nontrivial. We can arbitrarily increase the value of the objective function by raising off-diagonal terms of the matrix, but at some point this will violate positive semi-definiteness. Since positive semi-definiteness can be rephrased as the requirement that $(\Greekmath 011A _{ij})=AA^{T}$ for some matrix $A$, we can rewrite the constrained maximization problem as follows:
Denote $\Greekmath 010B =\arccos r$. Since the solution to ((ref)) is invariant to a rotation of all vectors $a_{i}$ , we can set
without loss of generality. Note that $a_{1},a_{n}$ are both unit norm and have dot product $r$. Thus, we have eliminated the constraint $ a_{1}^{T}a_{n}=r$ and reduced the variables in the maximization problem to $ a_{2},...,a_{n-1}$.
Now consider some $k=2,...,n-1$. Fix $a_{j}$ for all $j\neq k$, and choose $ a_{k}$ to maximize the objective function. As a first step, we show that $ a_{k}$ must be a linear combination of $a_{k-1},a_{k+1}$. To show this, we write $a_{k}=u+v$, where $u,v$ are orthogonal vectors, $u$ is in the subspace spanned by $a_{k-1},a_{k+1}$ and $v$ is orthogonal to the subspace. Recall that $a_{k}$ is a unit-norm vector, which implies that
The terms in the objective function ((ref)) that depend on $a_{k}$ are simply $(a_{k-1}^{T}u)(a_{k+1}^{T}u)$. All the other terms in the product do not depend on $a_{k}$, whereas the dot product between $a_{k}$ and $a_{k=1},a_{k+1}$ is invariant to $v$: $ a_{k-1}^{T}(u+v)=a_{k=1}^{T}u$.
Suppose that $v$ is nonzero. Then, we can replace $a_{k}$ with another unit-norm vector $u/\Vert u\Vert $, such that $(a_{k-1}^{T}u)(a_{k+1}^{T}u)$ will be replaced by
By ((ref)) and the assumption that $v$ is nonzero, $\Vert u\Vert <1$, hence the replacement is an improvement. It follows that $a_{k}$ can be part of an optimal solution only if it lies in the subspace spanned by $a_{k-1},a_{k+1}$. Geometrically, this means that $a_{k}$ lies in the plane defined by the origin and $a_{k-1},a_{k+1}$.
Having established that $a_{k},a_{k-1},a_{k+1}$ are coplanar, let $\Greekmath 010B $ be the angle between $a_{k}$ and $a_{k-1}$, let $\Greekmath 010C $ be the angle between $a_{k}$ and $a_{k+1}$, and let $\Greekmath 010D $ be the (fixed) angle between $a_{k-1}$ and $a_{k+1}$. Due to the coplanarity constraint, $\Greekmath 010B +\Greekmath 010C =\Greekmath 010D $. Fixing $a_{j}$ for all $j\neq k$ and applying a logarithmic transformation to the objective function, the optimal $a_{k}$ must satisfy
Differentiating this expression with respect to $\Greekmath 010B $ and setting the derivative to zero, we obtain $\Greekmath 010B =\Greekmath 010C =\Greekmath 010D /2$. Since this must hold for any $k=2,...,n-1$, we conclude that at the optimum, any $ a_{k}$ lies on the plane defined by the origin and $a_{k-1},a_{k+1}$ and is at the same angular distance from $a_{k-1},a_{k+1}$. That is, an optimum must be a set of equiangular unit vectors on a great circle, equally spaced between $a_{1}$ and $a_{n}$. The explicit formulas for these vectors are given by ((ref)).
The formula for the upper bound has a simple geometric interpretation (illustrated by Figure 2). We are given two points on the unit $n$ -dimensional sphere (representing $a_{1}$ and $a_{n}$) whose dot product is $ r$, and we seek\ $n-2$ additional points on the sphere such that the harmonic average of the successive points' dot product is maximal. Since the dot product for points on the unit sphere decreases with the spherical distance between them, the problem is akin to minimizing the average distance between adjacent points. The solution is to place all the additional points equidistantly on the great circle that connects $a_{1}$ and $a_{n}$.
Since by construction, every neighboring points $a_{k}$ and $a_{k+1}$ have a dot product of $\cos \Greekmath 0112 _{r,n}$, we have $\Greekmath 011A _{k,k+1}=\cos \Greekmath 0112 _{r,n}$, such that $\hat{\Greekmath 011A }_{1n}=(\cos \Greekmath 0112 _{r,n})^{n-1}$. This completes the proof.
This paper performed a worst-case analysis of misspecified recursive models. We showed that within this class, model selection is a very powerful tool in the hands of an opportunistic analyst: If we allow him to freely select a moderate number of variables from a large pool, he can produce a very large estimated correlation between two variables of interest. Furthermore, the structure of his model allows him to interpret this correlation as a causal effect. This is true even if the two variables are objectively independent, or if their correlation is in the opposite direction. Imposing a bound on the model's complexity (measured by its number of auxiliary variables) is an important constraint on the analyst. However, the value of this bound decays quickly, as even with one or two auxiliary variables the analyst can greatly distort objective correlations.
Within our framework, several questions are left open. First, we do not know whether Theorem (ref) would continue to hold if we replaced the quantifier \textquotedblleft for almost every $p$\textquotedblright\ with \textquotedblleft for every $p$\textquotedblright . Second, we do not know how much bite the undistorted-variance constraint has in models with more than one non-trivial equation. Third, we lack complete characterizations for recursive models outside the linear-regression family (see our partial characterization for models that involve binary variables in Appendix II). Finally, it would be interesting to devise a sparse collection of misspecification or robustness tests that would restrain our opportunistic analyst.
Taking a broader perspective into the last question, our exercise suggests a novel approach to the study of biased estimates due to misspecified models in Statistics and Econometric Theory (foreshadowed by Spiess (2018)). Under this approach, the analyst who employs a structural model for statistical or causal analysis is viewed as a player in a game with his audience. Researcher bias implies a conflict of interests between the two parties. This bias means that the analyst's model selection is opportunistic. The question is which strategies the audience can play (in terms of robustness or misspecification tests it can demand) in order to mitigate errors due to researcher bias, without rejecting too many valuable models.
{ Appendix: Uniform Binary Variables}
Suppose now that the variables $x_{1},...,x_{n}$ all take values in $\{-1,1\}$, and restrict attention to the class of objective distributions $p$ whose marginal on each variable is uniform - i.e., $ p(x_{i}=1)=\frac{1}{2}$ for every $i=1,...,n$. As in our main model, fix the correlation between $x_{1}$ and $x_{n}$ to be $r$ - that is,
The question of finding the distribution $p$ (in the above restricted domain) and the DAG $G$ that maximize the induced $\hat{\Greekmath 011A }_{in}$ subject to $p_{G}(x_{n})=\frac{1}{2}$ is generally open. However, when we fix $G$ to be the linear DAG
we are able to find the maximal $\hat{\Greekmath 011A }_{1n}$. It makes sense to consider this specific DAG, because it proved to be the one most conducive to generating false correlations in the case of linear-regression models.
Given the DAG $G$ and the objective distribution $p$, the correlation between $x_{i}$ and $x_{j}$ that is induced by $p_{G}$ is
Let $j>i$. Given the structure of the linear DAG, we can write
In particular,
Note that $p_{G}(x_{n}\mid x_{2})$ has the same expression that we would have if we dealt with a linear DAG of length $n-1$, in which $2$ is the ancestral node: $2\rightarrow \cdots \rightarrow n$. This observation will enable us to apply an inductive proof to our result.
We can now derive an upper bound on $\hat{\Greekmath 011A }_{in}$ for the environment of this appendix - i.e., the estimated model is a linear DAG, and the objective distribution has uniform marginals over binary variables.$\medskip $
How does this upper bound compare with the Gaussian case? For illustration, let $r=0$. Then, it is easy to see that for $n=3$, we obtain $\hat{\Greekmath 011A } _{13}=\frac{1}{3}$, which is below the value of $\frac{1}{2}$ we were able to obtain in the Gaussian case. And as $n\rightarrow \infty $, $\hat{\Greekmath 011A } _{1n}\rightarrow 1/e$. That is, unlike the Gaussian case, the maximal false correlation that the linear DAG can generate is bounded far away from one.
The upper bound obtained in this result is tight. The following is one way to implement it. For the case $r=0$, take the exact same Gaussian distribution over $x_{1},...,x_{n}$ that we used to implement the upper bound in Theorem (ref), and now define the variable $y_{k}=sign(x_{k})$ for each $k=1,...,n$. Clearly, each $y_{k}\in \{-1,1\}$ and $ p(y_{k}=1)=p(y_{k}=-1)=\frac{1}{2}$ since each $x_{k}$ has zero mean. To find the correlations between different $y_{k}$ variables, we use the following lemma.
Returning to the definition of the Gaussian distribution over $ x_{1},...,x_{n}$ that we used to implement the upper bound in Theorem (ref), we see that in the case of $r=0$, the angle between $w_{1}$ and $ w_{n}$ will be $\frac{\Greekmath 0119 }{2}$, so that by the above lemma, $y_{1}$ and $ y_{n}$ will be uncorrelated. At the same time, the angle between any $w_{k}$ and $w_{k-1}$ is by construction $\frac{\Greekmath 0119 }{2}\frac{1}{n-1}$ because the vectors were chosen at equal angles along the great circle. Substituting this angle into the lemma, we obtain that the correlation between $y_{k}$ and $y_{k-1}$ is $1-\frac{1}{n-1}$.
For the case where $r\neq 0$, the same argument holds, except that we need to choose the original vectors $w_{1},w_{n}$ so that the correlation between $y_{1}$ and $y_{n}$ will be $r$ (these will not be the same vectors that give a correlation of $r$ between the Gaussian variables $x_{1}$ and $x_{n}$ ) and then choose the rest of the vectors at equal angles along the great circle. By applying the lemma again, we obtain that the angle between $y_{k}$ and $y_{k-1}$ is $1-\frac{1-r}{n-1}$, which again attains the upper bound.
This method of implementing the upper bound also explains why false correlations are harder to generate in the uniform binary case, compared with the case of linear-regression models. The variable $y_{k}$ is a coarsening of the original Gaussian variable $x_{k}$. It is well-known that when we coarsen Gaussian variables, we weaken their mutual correlation. Therefore, the correlation between any consecutive variables $y_{k},y_{k+1}$ in the construction for the uniform binary case is lower than the corresponding correlation in the Gaussian case. As a result, the maximal correlation that the model generates is also lower.
The obvious open question is whether the restriction to linear DAGs entails in a loss of generality. We conjecture that in the case of uniform binary variables, a non-linear perfect DAG can generate larger false correlations for sufficiently large $n$.