Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
53,279 characters · 14 sections · 39 citation commands
Optimal Text-Based Time-Series Indices
\doublespacing
In economic and financial research, there is a growing trend of integrating textual data such as news articles into econometrics analysis gentzkow2019text. This integration is typically done by (i) selecting, (ii) transforming, and (iii) aggregating textual content into a time-series representation ArdiaEtAl2019,algaba2020econometrics. While many studies have focused on steps (ii) and (iii)---transforming and aggregating textual data into a quantitative measure such as sentiment loughran2014measuring,jegadeesh2013word,manela2017news---the essential selection step (i), which usually relies on subjective ad-hoc rules, has not received much attention yet.
We aim to fill this gap in this article by proposing an approach to construct text-based time-series indices optimally. Specifically, our algorithm determines which set of texts, among a large corpus, leads to a text-based index that is optimal for a specific objective---typically, an index that maximizes the contemporaneous relation or the predictive performance with respect to a target variable, such as inflation. Our methodology relies on binary selection matrices that, applied to the vocabulary of tokens, select the relevant texts in the corpus. Various widely-known text-based indices, such as the Economic Policy Uncertainty (EPU) index by BakerEtAl2016, can be formulated in terms of selection matrices.
Optimizing selection matrices is challenging due to the inherent non-linearity that arises when aggregating selected texts into text-based indices. To overcome this difficulty, we design a genetic algorithm with domain-specific knowledge to explore the solution space and obtain the optimal selection matrix. The algorithm starts with an initial population of selection matrices. These matrices are evaluated using a fitness function that measures the resulting textual-index performance in achieving the objective of selecting the texts. At each iteration, the population of selection matrices undergoes tailor-made crossover and mutation operations, as traditional operators are not well suited to this optimization problem. We also implement additional steps to address potential overfitting issues and leverage the information in word embeddings of promising solutions to help explore good solutions more efficiently. Finally, we introduce a pruning step to avoid sub-optimal solutions.
To showcase the relevance of our methodology, we conduct two empirical applications using a collection of 763,542 Wall Street Journal news articles spanning from January 2000 to August 2021.
First, we validate our methodology with the EPU index. Specifically, we use our algorithm to see if we can recover the set of keywords proposed by BakerEtAl2016 when building their EPU index. Given the number of tokens in our corpus vocabulary, recovering the set of tokens for the three dictionary dimensions of BakerEtAl2016 is not trivial. We show that our approach (i) can recover the set of keywords proposed by the authors---their selection matrix---and (ii) does so in a reasonable computational time.
Second, we evaluate the performance of our methodology against established benchmarks, focusing on the monthly VIX index and inflation expectations derived from the Michigan Surveys of Consumers. In this comparative analysis, the text-based indices generated by our methodology consistently outperform the text-based benchmarks on the test window, a time period not considered during the optimization of the selection matrices.
The rest of this paper is organized as follows. Section (ref) introduces the notation, presents the concept of selection matrices, and outlines the optimization problem. Section (ref) presents our optimization strategy, including the genetic algorithm as well as the proposed crossover and mutation operators. Section (ref) presents our empirical applications, and Section (ref) concludes.
We first introduce the concept of tokens and vocabulary to analyze a corpus of texts. A token, denoted by $v$, represents a sequence of characters, including acronyms, words, sequences of words, or even regular expressions. The vocabulary of size $V$ is defined as a collection of such tokens.
The text corpus is represented as a matrix $\bm{C}_t$ of size $N_t \times V$, where $N_t$ is the number of texts available at a given time $t$. Each element $c_{n,v,t}$ among a collection of matrices $\bm{C}_t$, where $t = 1,\dots,T$, indicates if the token $v$ appears in the text $n$ published at time $t$. If it does, the element takes the value of one and zero otherwise. Hence, a specific text published at time $t$ can be represented by the row vector $\bm{c}_{n,t}$ (of size $1 \times V$).
Typically, the corpus consists of a vast collection of texts, and our objective is to select texts for further analysis. For instance, we could focus on texts related to the U.S. economy from a collection of news articles published by various newspapers. A simple way to proceed is by using a keyword-based (i.e.\xspace, token-based in our nomenclature) approach, which we formalize below.
We employ a selection matrix $\bm{\Omega}$ of size $V \times K$, where each column vector $\bm{\omega}^k$ (of size $V \times 1$) corresponds to a selection rule. The binary selection vector $\bm{\kappa}_t$ of size $N_t$ contains elements $\kappa_{n,t}$, which are defined through the selection function $f_\kappa(\cdot)$ as:
where $I[\cdot]$ is an indicator function that takes a value of one if the condition inside the parenthese is true and zero otherwise.
In (ref), $\sum_{k=1}^{K} I\big[\bm{c}_{n,t} \bm{\omega}^k \ge 0\big]$, counts the number of unique active tokens (i.e.\xspace, non-zero value) for dimension $k$ of the selection matrix $\bm{\Omega}$ (represented by $\bm{\omega}^k$) that appear in the text $n$ at time $t$ (represented by row vector $\bm{c}_{n,t}$). If this count is greater than zero for all dimensions $k=1,\ldots,K$, each element of the sum takes the value of one, and the summation takes the value $K$. The second part, $\sum_{k = 1}^{K}I\big[\sum_{v = 1}^{V}\bm{i}_V \bm{\omega}^k > 0\big]$, where $\bm{i}_V$ is a row vector of lenght $V$ where each element is equal to one, counts the number of active tokens within each dimension of the selection matrix $\bm{\Omega}$. If this count is greater than zero for all dimensions, each element of the sum takes value one, and the sum is $K$. Thus, if the text $n$ published at time $t$ contains at least one active token within each non-zero column $k$ of $\bm{\Omega}$, the selection condition is satisfied, and $\kappa_{n,t}$ is equal to one.
The well-known Economic Policy Uncertainty (EPU) index of BakerEtAl2016 relies on this selection function. The EPU selection condition is defined by a vocabulary of size 10 with three selection dimensions ($V = 10$ and $K = 3$) representing economic, policy, and uncertainty-related tokens. Using our notation, the selection matrix $\bm{\Omega}$ (of size $10 \times 3$) of the EPU index is: $$ \bm{\Omega}_{\text{EPU}} \equiv \kbordermatrix{ & \text{Economy} & \text{Policy} & \text{Uncertainty} \\ \text{economic} & 1 & 0 & 0 \\ \text{economy} & 1 & 0 & 0 \\ \text{congress} & 0 & 1 &0 \\ \text{deficit} & 0 & 1 & 0\\ \text{federal\_reserve} & 0 & 1 & 0\\ \text{legislation} & 0 & 1 & 0\\ \text{regulation} & 0 & 1 & 0\\ \text{white\_house} & 0 & 1 & 0\\ \text{uncertain} & 0 & 0 & 1 \\ \text{uncertainty} & 0 & 0 & 1 \\ } \,.$$ The EPU selection matrix contains 2, 6, and 2 active tokens in the first, second, and third dimension, respectively. In that case, the second terms of (ref): $\sum_{k = 1}^{K}I\big[\sum_{v = 1}^{V}\bm{i}_V \bm{\omega}^k_{\text{EPU}} > 0\big] = 3$. For a text $n$ published at time $t$ containing the tokens “economic,” “congress,” and “uncertainty,” the first terms of (ref): $\sum_{k=1}^{K} I\big[\bm{c}_{n,t} \bm{\omega}^k_{\text{EPU}} \ge 0\big] = 3$. Given the equality of both terms, for this article, $\kappa_{n,t} = 1$. Alternatively, a text $n$ published at time $t$ containing two times the token “economic” and one time the token “congress,” the first terms of (ref), $\sum_{k=1}^{K} I\big[\bm{c}_{n,t} \bm{\omega}^k_{\text{EPU}} \ge 0\big] = 2$, and thus $\kappa_{n,t} = 0$ as both the left and right terms of (ref) are not equal.
Several news-media-based indices follow similar selection rules: the Climate Policy Uncertainty index of gavriilidis2021measuring, the Equity Market Volatility index of baker2019policy, the Monetary Policy Uncertainty index of husted2020monetary, and the Trade Policy Uncertainty index of handley2022trade.
The selection function in (ref) does not, however, take into account more complex selection mechanisms, such as the addition of a proximity condition between the tokens of two or more dimensions of the selection matrix $\bm{\Omega}$, as the one used to construct the Geopolitical Risk index of caldara2022measuring. For instance, in the case of the EPU index, one might require at least one token from the “Economy” dimension detected in the neighborhood (e.g.\xspace, within a two-token distance) of one of the “Policy” tokens in the same news article. Such additional complexity in the selection rule could be integrated using the information from a token distance matrix similar to the one used in ardia2021century. To keep the exposition simple, we do not consider this type of variation of the selection function $f_\kappa(\cdot)$ in this paper.
After identifying relevant texts in the corpus with the selection function $f_\kappa(\cdot)$, we typically transform and aggregate the selected texts into a quantitative time-series measure. We present below attention-based and context-based time-series measures.
An attention measure aims to measure the level of importance attributed to the selected texts across all texts in the corpus at a given time. For instance, it could be to measure the level of importance given to the topic of climate change by the news media ArdiaEtAl2023. Given our prior notation and definition, a typically used measure of attention is:
At each time $t$, we measure how many texts are selected and normalize this number by the total number of texts in the corpus at that time. If the corpus is composed of several sources (e.g.\xspace, various newspapers), it may be relevant to standardize each source before the aggregation in (ref), as in BakerEtAl2016.
An alternative form of transformation is the content transformation, which requires more information than the output of the selection step. In particular, the content of selected texts is processed to derive a “score,” such as polarity or sentiment algaba2020econometrics. The content transformation function can be written as:
where $\zeta_v$ are token weights (integer or real numbers) used to score the selected texts. The average score of selected texts at time $t$ is then used to obtain an index. Token weights can be measured from manually-composed lexicons loughran2014measuring or in a data-driven way jegadeesh2013word,manela2017news,kelly2021text,ardia2022media. Much of the literature has focused on estimating $\bm{\zeta} \equiv (\zeta_1,\ldots,\zeta_V)$ given a predefined selection of texts.
A practical advantage of employing the attention transformation over the content transformation lies in its reduced reliance on processing the entire corpus of texts through an additional layer of models to derive $\bm{\zeta}$ algaba2020econometrics. For instance, when utilizing a news articles aggregator that consolidates the desired news sources, the data required for computing attention solely invoke queries derived from a selection matrix (to determine the count of selected articles) and statistical information on the number of published news articles from the desired sources. This streamlined approach facilitates the timely computation and updating of the attention index. See, for instance, https://www.policyuncertainty.com/, where multiple indices similar to the EPU are updated monthly.
When using a data-driven method for content transformation, the joint estimation of the selection matrix $\bm{\Omega}$, and content transformation weights $\bm{\zeta}$, is preferable, albeit computationally intensive. Because of the added complexity of the content transformation, which is a challenge in this study, to isolate the effect of news selection from content transformation, we focus on the attention transformation measure.
The objective of deriving information in texts by a selection function and a transformation process is to capture a (contemporaneous or predictive) relation between that information and a variable of interest $y_t$ $(t=1,\ldots,T)$. A selection matrix is typically based on a subjective but guided assessment of the tokens necessary to capture the information we want to extract from the texts. Formally, given a relevant transformation function $f(\bm{\Omega}, \bm{C}_t)$, the aim is to minimize:
with $h\geq0$, which is a simple linear regression problem.\footnote{While we present here the case of a linear relationship between $y_t$ and $f(\bm{\Omega}, \bm{C}_{t-h})$, more general specifications would work just as well, including multiple linear regression model by adding additional explanatory variables or by not assuming a linear relationship. To ease the presentation, we focus on a simple linear relationship without additional explanatory variables.} Once estimated, one would typically make inference about parameter $\beta$ or test whether the (out-of-sample) root-mean-squared error is significantly lower than the root-mean-squared error of a model where $\beta$ is constrained to be zero (i.e.\xspace, nested model); see ArdiaEtAl2019. For instance, several studies analyze the relationship between the Economic Policy Uncertainty and economic variables using the following regression framework:\footnote{At the time of this writing, for instance, BakerEtAl2016, the research paper introducing the EPU index, has received over 9,000 citations according to Google Scholar. While many of those citations do not use the index per se, many, however, analyze how the EPU explains or interacts with other macroeconomics and financial variables.}
One limitation of this approach is related to the fact that $\bm{\Omega}_\text{EPU}$ is given. This may be reasonable in some applications but suboptimal for others, particularly when it comes to nowcasting and forecasting. We thus introduce the following optimization problem:
where $\bm{\omega}_v$ denotes the $v$th row of $\bm\Omega$. In optimization problem (ref), we assume an unrestrictive and large vocabulary and optimize active tokens selection in addition to the regression parameters. We use three penalty terms to control the optimization process. We standardize $y_t$ to avoid scale-dependent penalty parameters.
First, we introduce a penalty term $\lambda_1 \sum_{v = 1}^{V}I[ \bm{\omega}_v \bm{i}'_K > 1]$, which aims to avoid the activation of the same token across multiple dimensions in the selection matrix. A higher value of $\lambda_1$ enforces a constraint that restricts each dimension from overlapping. This constraint can be beneficial when assigning specific topics to each dimension. However, it also limits the flexibility of the selection matrix. Consider the following two three-dimensional selection matrices: $$ \kbordermatrix{ & \text{1} & \text{2} & \text{3} \\ \text{economic} & 1 & 1 & 1 \\ \text{stock\_market} & 1 & 0& 0\\ \text{federal\_reserve} & 0 & 1 &0 \\ \text{crisis} & 0 & 0 & 1 \\ } \quad \quad \kbordermatrix{ & \text{1} & \text{2} & \text{3} \\ \text{economic} & 1 & 0 & 0 \\ \text{stock\_market} & 1 & 0& 0\\ \text{federal\_reserve} & 0 & 1 &0 \\ \text{crisis} & 0 & 0 & 1 \\ } $$
With the first selection matrix, a text containing the token “economic” would be selected, while it would not necessary with the second, as in this case, a text needs to include tokens from the other dimensions, distinct from “economic.” Therefore, a lower value of $\lambda_1$ allows for the inclusion of selection conditions with lower dimensions than the one of the selection matrix, meaning, for instance, a text may fulfill the requirement with a single distinct token instead of three distinct tokens (when $K = 3$). Setting $\lambda_1 = 0$ provides an opportunity for a higher-dimensional selection matrix to mimic a lower-dimensional one, as shown by these two equivalent solutions: \[ \bm{\Omega} = \kbordermatrix{ & \text{1} & \text{2} & \text{3} \\ \text{economic} & 1 & 1 & 1 \\ }= \kbordermatrix{ & \text{1} \\ \text{economic} & 1 \\ } \]
Increasing flexibility with $\lambda_1 = 0$ increases complexity, which may lead to challenges in the selection matrix's out-of-sample performance. A higher value of $\lambda_1$ penalizes the score of the left selection matrix and, therefore, leads to a narrower set of potential solutions.
The two remaining penalty terms, $\lambda_2 I[\beta > 0]$ and $\lambda_3 I[\beta \leq 0]$, serve to establish the intended association between the text-based index and the variable of interest $y_t$. When a negative relationship is sought, the condition $\lambda_2 > \lambda_3$ is applied. Consequently, this allows for controlling a selection condition that selects texts either positively or negatively correlated with the variable of interest. It is crucial to choose the sign of beta as selection matrices can be derived in a manner that selects texts associated negatively with the variable of interest and another set that is positively associated with it.
Regression (ref) is trivial to estimate as $\bm{\Omega}$ is fixed and thus not part of the estimation process. On the contrary, the minimization problem (ref) is complex as $f(\bm{\Omega}, \bm{C}_t)$ is a non-linear function of $\bm{\Omega}$, which is now considered as a parameter. Below, we present our strategy based on a genetic algorithm to perform the minimization. We first define the crossover and mutation operators specifically designed for our problem.
Because of the distinctive characteristics of our optimization problem, we introduce specific crossover and mutation operators. To illustrate these operators, we consider a specific scenario wherein our optimization objective is to replicate the EPU index while imposing a constraint that restricts the number of active tokens to five out of a maximum of $V\!K$ (i.e.\xspace, when all elements in the matrix $\bm{\Omega}$ are equal to one).
We begin with the “token crossover operator.” This operator takes two parent solutions and generates two offspring solutions by exchanging a single token between the parents while maintaining the dimensionality of the replaced token. Consider the following two parents: $$ \kbordermatrix{ \text{Parent 1:} & 1 & 2 & 3 \\ \colorbox{gray!40}{\text{economic}} & 1 & 0 & 0 \\ \text{legislation} & 0 & 1 & 0 \\ \text{congress} & 0 & 1 & 0 \\ \text{house} & 0 & 0 & 1\\ \text{risk} & 0 & 0 & 1 \\ \vdots & \vdots & \vdots &\vdots } \quad\quad \kbordermatrix{ \text{Parent 2:} & 1 & 2 & 3 \\ \text{economy} & 1 & 0 & 0 \\ \colorbox{gray!20}{\text{financial\_crisis}} &0 & 1 & 0 \\ \text{congress} & 0 & 1 & 0 \\ \text{house} & 0 & 0 & 1\\ \text{risk} & 0 & 0 & 1 \\ \vdots & \vdots & \vdots &\vdots } $$
The token crossover operator selects two tokens (in gray) and generates two offspring solutions as follows: $$ \kbordermatrix{ \text{Child 1} & 1 & 2 & 3 \\ \colorbox{gray!20}{\text{financial\_crisis}} & 1 & 0 & 0 \\ \text{legislation} & 0 & 1 & 0 \\ \text{congress} & 0 & 1 & 0 \\ \text{house} & 0 & 0 & 1\\ \text{risk} & 0 & 0 & 1 \\ \vdots & \vdots & \vdots &\vdots } \quad \quad \kbordermatrix{ \text{Child 2} & 1 & 2 & 3 \\ \text{economy} & 1 & 0 & 0 \\ \colorbox{gray!40}{\text{economic}} &0 & 1 & 0 \\ \text{congress} & 0 & 1 & 0 \\ \text{house} & 0 & 0 & 1\\ \text{risk} & 0 & 0 & 1 \\ \vdots & \vdots & \vdots &\vdots } $$ It is important to note that this crossover operator maintains a fixed number of active tokens within each dimension and operates solely on already activated tokens. In contrast, a regular crossover operator would not guarantee that each dimension remains the same size or even has active tokens. Additionally, owing to the high sparsity of our optimization problem (where only five tokens are active out of a potential $V\!K$ points), a regular crossover operator could perform non-altering operations by exchanging slices of non-active tokens (i.e.\xspace, tokens with zero values), consequently reducing the efficiency of the search algorithm.
Next, we focus on three mutation operators, each serving a distinct purpose.
\paragraph{Switch Mutation Operator} The “switch mutation” operator allows an active token to change its dimension: $$ \kbordermatrix{ \text{Parent} & 1 & 2 & 3 \ \\ \text{economy} & 1 & 0 & 0 \\ \text{economic} &0 & \colorbox{gray!40}{1} & 0 \\ \text{congress} & 0 & 1 & 0 \\ \text{house} & 0 & 0 & 1\\ \text{risk} & 0 & 0 & 1 \\ \vdots & \vdots & \vdots &\vdots } \quad \quad \kbordermatrix{ \text{Child} & 1 & 2 & 3 \\ \text{economy} & 1 & 0 & 0 \\ \text{economic} &\colorbox{gray!40}{1} & 0 & 0 \\ \text{congress} & 0 & 1 & 0 \\ \text{house} & 0 & 0 & 1\\ \text{risk} & 0 & 0 & 1 \\ \vdots & \vdots & \vdots &\vdots } $$
This operator allows for the correction of potential dimensional misalignment. It is worth noting that in the example above, the set of texts selected by the parent, while not optimal, may be highly correlated with the set of texts of the child, which is optimal. This operation facilitates a more efficient search for misalignment compared to a random search. Similar to other operators, this mutation operator only applies to active tokens. However, it cannot change the dimensions of a token if it is the only token within its dimension, ensuring that each dimension has at least one active token.
\paragraph{N-Gram Mutation Operator}
Next, we introduce the “n-gram mutation” operator, which involves transforming one of the selected tokens into an n-gram containing that token: $$ \kbordermatrix{ \text{Parent} & 1 & 2 & 3 \\ \text{economy} & 1 & 0 & 0 \\ \text{economic} & 1 & 0 & 0 \\ \text{congress} & 0 & 1 & 0 \\ \colorbox{gray!40}{\text{house}} & 0 & 0 & 1\\ \text{risk} & 0 & 0 & 1 \\ \vdots & \vdots & \vdots &\vdots } \quad \quad \kbordermatrix{ \text{Child} & 1 & 2 & 3 \\ \text{economy} & 1 & 0 & 0 \\ \text{economic} & 1 & 0 & 0 \\ \text{congress} & 0 & 1 & 0 \\ \colorbox{gray!40}{\text{white\_house}} & 0 & 0 & 1\\ \text{risk} & 0 & 0 & 1 \\ \vdots & \vdots & \vdots &\vdots } $$
To comprehend this operator, we must recognize that the texts containing an n-gram always include the texts containing its components. For instance, the number of texts containing the token “stock” is equal to or larger than the number of texts containing the bi-gram “stock_market.” As such, if the algorithm selects the component of an n-gram, there is a high likelihood that using an n-gram containing that component would be more optimal. The n-gram mutation operator increases the possibility of testing an n-gram compared to testing any other token. Similar to the crossover operator, the n-gram mutation only applies to active tokens, but it also applies solely to tokens that are part of n-grams within the vocabulary.
\paragraph{Transform Mutation Operator} Finally, we have the “transform mutation” operator, which changes a token from a specific dimension to another token in that same dimension: $$ \kbordermatrix{ \text{Parent} & 1 & 2 & 3 \\ \text{economy} & 1 & 0 & 0 \\ \text{economic} & 1 & 0 & 0 \\ \text{congress} & 0 & 1 & 0 \\ \text{white\_house} & 0 & 0 & 1\\ \colorbox{gray!40}{\text{risk}} & 0 & 0 & 1 \\ \vdots & \vdots & \vdots &\vdots } \quad \quad \kbordermatrix{ \text{Child} & 1 & 2 & 3\\ \text{economy} & 1 & 0 & 0 \\ \text{economic} & 1 & 0 & 0 \\ \text{congress} & 0 & 1 & 0 \\ \text{white\_house} & 0 & 0 & 1\\ \colorbox{gray!40}{\text{uncertainty}} & 0 & 0 & 1 \\ \vdots & \vdots & \vdots &\vdots } $$ The transform mutation is the only operator that activates new tokens (not n-grams) that have not already been activated within the population. This operator thus introduces new tokens within the population by replacing an active token from a candidate selection matrix with a new one. The transform mutation ensures that the number of active tokens within each dimension remains the same. This operator is crucial in controlling the number of active tokens while allowing the introduction of new active tokens. A standard mutation operator that flips one bit would not work in this case. Consider a vocabulary of size 4,800 with three dimensions, leading to $4,\!800 \times 3 = 14,\!400$ potential number of active points. If a parent has ten active tokens out of 14,\!400, the standard mutation operator has a $\frac{14,\!400-10}{14,\!400})= 99.93\%$ chance of generating a child with 11 active tokens and a $0.07\%$ chance of generating a child with nine active tokens. Over a large number of iterations, this leads to a search space with a large number of active tokens, ultimately leading to overfitting.
By design, many of these operators only apply to tokens that are already active within the corpus to ensure an efficient search. Due to the extreme sparsity of our optimization problem, without these operators, the likelihood of performing non-altering operations is extremely high. We also note that these operations ensure that the number of active tokens in the child will always be the same as the number of active tokens in the parent. This is crucial for our search strategy, which is presented below.
The calibration strategy is centered around (i) efficient search and (ii) avoiding overfitting. First, we define a training and a validation window. We start the optimization process by defining the initial population of $q = 1, \dots, Q$ candidate selection matrices $\bm{\Omega}_q$ of size $V \times K$. Each member of the initial population starts with $K$ active tokens (non-zero value in the selection matrix), where $K$ is the number of dimensions of the selection matrix. Each dimension is set to have one active token. For instance, for $K = 3$, these could be members of the initial population (displaying only active tokens): \[ \kbordermatrix{ & 1 & 2 & 3 \ \\ \text{democrats} & 1 & 0 & 0 \\ \text{european\_union} & 0 & 1 & 0 \\ \text{policy} & 0 & 0 &1 \\ \vdots & \vdots & \vdots &\vdots } \quad\quad \kbordermatrix{ & 1 & 2 & 3 \ \\ \text{religion} & 1 & 0 & 0 \\ \text{market} & 0 & 1 & 0 \\ \text{presidency} & 0 & 0 &1 \\ \vdots & \vdots & \vdots &\vdots } \] while these would not:
\[ \kbordermatrix{ & 1 & 2 & 3 \ \\ \text{democrats} & 1 & 0 & 0 \\ \text{european\_union} & 0 & 0 & 1 \\ \text{policy} & 0 & 0 &1 \\ \vdots & \vdots & \vdots &\vdots } \quad\quad \kbordermatrix{ & 1 & 2 & 3 \ \\ \text{religion} & 1 & 0 & 0 \\ \text{market} & 0 & 1 & 0 \\ \text{presidency} & 0 & 0 &1 \\ \text{coffee} & 0 & 0 &1 \\ \vdots & \vdots & \vdots &\vdots } \]
Inpired from Scrucca2013,Scrucca2017, we perform genetic optimization as follows:
The solution with the lowest objective value from the validation window from step 2 of the last epoch is considered as the interim optimal solution. To obtain the final solution, we perform a pruning step. This step aims to eliminate tokens that may have adversely affected the model fit but were added due to the random nature of the optimization process (e.g.\xspace, uninformative tokens added before informative ones). To accomplish this, we test variations of the optimal selection matrix by deactivating one or several active tokens, encompassing all variations involving only deactivations. For instance, if the solution with the lowest objective value from the validation window has 12 active tokens, we test $2^{12}= 4,\!096$ potential new solutions (ranging from all tokens being active to all tokens being deactivated).
In this section, we present two empirical applications of our methodology. First, we validate our approach with the EPU index. Second, we apply our optimization process to (i) tracking the VIX and (ii) nowcasting inflation expectations. We use the latter two usecases as news-based indices have been used in previous studies in relation to those variables, and as such, we can test our method against benchmarks. Beforehand, we describe the corpus of text used throughout this section and the data processing steps.
Our corpus consists of all news articles published in the printed version of the Wall Street Journal from January 2000 to August 2021. In total, the corpus is composed of 763,542 news articles. To derive candidate tokens for the vocabulary describing the corpus, we proceed as in ArdiaEtAl202x:
We use the pre-trained continuous bag-of-word embedding model of rahimikia2021realised available at https://fintext.ai to retrieve the token vector for each of the 2,261 tokens. The model is compiled from 15 years of business news archives and, as a domain-specific model, is more appropriate to capture the relationship between tokens than a pre-trained general-domain word embedding model such as Google Word2Vec. For collocations, we take the average of the token vector across the individual collocation tokens. These token vector representations of the 2,261 tokens will be used to narrow down the vocabulary after each epoch.
Our first application aims to validate our approach by determining if we can recover the EPU selection matrix. Specifically, we will perform the following optimization problem:
and see if $\hat{\bm{\Omega}} = \bm{\Omega_{\text{EPU}}}$. With $V = 2,\!261$ and $K = 3$, the number of selection matrices is $2^{2261\times 3}$. Convergence toward the true EPU selection matrix would thus indicate massive improvement compared to a naive random search.
The setup for this experiment is as follows. We set the size of the selection matrices population to $Q = 300$, the number of iterations per epoch to $H = 200$, and the number of epochs to 15. The training window is composed of monthly data from January 2000 to December 2010 (132 observations), and the validation window ranges from January 2011 to December 2014 (48 observations). The remaining data from January 2015 to August 2021 is our test window (80 observations). We set $\lambda_1 = 0.25$, $\lambda_2 = 0$, and $\lambda_3 = 1$, so that we force the algorithm to search for tokens that have a positive relationship to the dependent variable. We test our algorithm for selection matrices with dimensions $K = 1,2,3$.
The optimal selection matrices obtained for each dimension are shown below: \[ \hat{\bm{\Omega}}_{K = 1} \!=\!\!\! \!\!\kbordermatrix{ & 1 \ \\ \text{uncertainty} & 1 \\ \text{uncertain} & 1 \\ \text{loom} & 1 \\ \text{unknown} & 1 \\ \vdots & \vdots } \quad\!\! \hat{\bm{\Omega}}_{K = 2} \!=\!\!\!\!\!\kbordermatrix{ & 1 & 2 \ \\ \text{uncertainty} & 1 & 0 \\ \text{uncertain} & 1 & 0 \\ \text{monetary} & 0 & 1 \\ \text{banking} & 0 & 1 \\ \text{regulate} & 0 & 1 \\ \text{economic\_growth} & 0 & 1 \\ \text{spending} & 0 & 1 \\ \text{overhaul} & 0 & 1 \\ \text{extend} & 0 & 1 \\ \text{necessary} & 0 & 1 \\ \text{reserve} & 0 & 1 \\ \text{democrats} & 0 & 1 \\ \vdots & \vdots & \vdots } \quad \hat{\bm{\Omega}}_{K = 3} \!= \!\!\!\!\!\kbordermatrix{ & 1 & 2 & 3 \ \\ \text{uncertainty} & 1 & 0 & 0\\ \text{uncertain} & 1 & 0 & 0 \\ \text{deficit} & 0 & 1 & 0 \\ \text{congress} & 0 & 1 & 0 \\ \text{legislation} & 0 & 1 & 0\\ \text{regulation} & 0 & 1 & 0 \\ \text{federal\_reserve} & 0 & 1 & 0 \\ \text{white\_house} & 0 & 1 & 0 \\ \text{economic} & 0 & 0 & 1 \\ \text{economy} & 0 & 0 & 1 \\ \vdots & \vdots & \vdots } \]
We note that $\hat{\bm{\Omega}}_{K = 3}$ is exactly the EPU selection matrix, thus confirming that our algorithm can converge to an exact solution even with such a large search space.
In Table (ref), we report the number of active tokens and the training window, validation window, and test window root mean-squared-errors (RMSE) for each epoch, along with the optimal results after the pruning step. We notice that the reduction in all error measures is much less pronounced when the wrong number of dimensions of the selection matrix is used. We also see that the pruning step was necessary for retrieving the exact EPU matrix when optimizing the three-dimensional selection matrix. Indeed, on the 13 epochs, the three-dimensional selection matrix had 11 active tokens, of which one was removed after pruning. Finally, we note that no active token is added for some epochs, but for subsequent epochs, more than one active token is added. This might be because a token is only more informative in the presence of another token and not by itself.
The computational time on a standalone computer (single Intel core i9 3.7GHz with 48 GB RAM) ranges from five hours and 25 minutes for $K=1$ to nine hours and 45 minutes for $K=3$. Overall, this validation exercise demonstrates that our optimization can recover pre-defined selection matrices within a reasonable amount of computational time.
Next, we test our methodology for (i) tracking the VIX index and (ii) nowcasting inflation expectations. We use those two usecases because news articles were previously used to build text-indices to forecast these variables. Thus, we have natural benchmarks to compare the performance of our approach.
For the VIX, baker2019policy construct the Equity Market Volatility (EMV) index, which, in our framework, can be defined with the following selection matrix:
\[ \bm{\Omega}_{\text{EMV}} \equiv \kbordermatrix{ & \text{Equity} & \text{Market} & \text{Volatility} \\ \text{economic} & 1 & 0 & 0 \\ \text{economy} & 1 & 0 & 0 \\ \text{financial} & 1 & 0 &0 \\ \text{stock\_market} & 0 & 1 & 0\\ \text{equity} & 0 & 1 & 0\\ \text{equities} & 0 & 1 & 0\\ \text{standard\_and\_poors} & 0 & 1 & 0\\ \text{s\&p} & 0 & 1 & 0\\ \text{volatility} & 0 & 0 & 1 \\ \text{volatile} & 0 & 0 & 1 \\ \text{uncertain} & 0 & 0 & 1 \\ \text{uncertainty} & 0 & 0 & 1 \\ \text{risk} & 0 & 0 & 1 \\ \text{risky} & 0 & 0 & 1 \\ } \]
This news selection is then further analyzed and divided to quantify journalist perceptions about the news items, developments, concerns, and anticipations that drive volatility in equity returns. By optimizing the selection matrix, we aim to derive an attention index that more closely matches the VIX than the EMV does.\footnote{The VIX is retrieved from https://fred.stlouisfed.org/series/VIXCLS.} This is particularly useful as a better overarching index incorporates a more precise selection for selecting news having a positive relationship with market volatility, and thus better explains, in a quantitative manner (due to the availability of news data), the level of volatility. This could then be decomposed into a more narrow indicator as in baker2019policy.
Regarding inflation expectations, pfajfar2013news and marcellino2022demand analyze the link between news about inflation and how households form expectations about inflation. In particular, marcellino2022demand conclude that media communication and agents' attention play an important role in aggregate inflation expectations. In our framework, their attention index can be defined as:\footnote{In particular marcellino2022demand, search for instance of words containing “inflat” while pfajfar2013news search for words whose root is “inflation.” Since our words are lemmatized, all those instances are included in the word “inflation.”} $$ \bm{\Omega}_{\text{INFL}} \equiv \kbordermatrix{ & \text{inflation} \\ \text{inflation} & 1 \\ } $$
For a measure of inflation expectations, we use data from the Michigan University Survey of Consumers; we consider the median expected price change for the next 12 months (MICH).\footnote{MICH is retrieved from https://fred.stlouisfed.org/series/MICH.} Similar to the VIX, we want to find a selection matrix that generates a news attention index that more closely matches inflation expectations. This, in turn, can be used to determine better what news are related to inflation expectations by analyzing the tokens in the selection matrix or by further analyzing the news selected by the selection matrix (for instance, via a topic model).
The time-series indices constructed with $\bm{\Omega}_{\text{EMV}}$ and $\bm{\Omega}_{\text{INFL}}$ will serve as benchmarks when evaluating the fit on the test window. For the optimization, we use a population of size $Q = 200$ and $H = 500$ iterations per epoch with a maximum of 15 tokens (and, as such 13-15 epochs depending on the starting number of filtering dimensions). We set $\lambda_1 = 0.25$ and optimize for tokens positively correlated with the target index ($\lambda_2 = 0$ and $\lambda_3 = 1$). We keep the solution among $K = 1,2,3$ with the lowest objective value from the combined training and validation windows. Finally, we benchmark the results against $f_{\text{att}}(\bm{\Omega_{\text{EMV}}}, \bm{C}_t)$ and $f_{\text{att}}(\bm{\Omega_{\text{INFL}}}, \bm{C}_t)$ for the VIX and inflation expectations, respectivelly.
The best selection matrices obtained on the combined training and validation windows after pruning are shown below:
\[ \hat{\bm{\Omega}}_{\text{EMV}} =\!\!\!\! \kbordermatrix{ & & \\ \text{economy } & 1& 0 \\ \text{stock\_market } & 1 & 0 \\ \text{crisis} & 1 & 0 \\ \text{volatility} & 1& 0 \\ \text{despite} & 1 & 0 \\ \text{club} & 1& 0 \\ \text{fear} & 0 & 1 \\ \text{plunge} & 0 & 1 \\ \text{industrial\_average} & 0 & 1 \\ \text{package} & 0 & 1\\ \text{summit} & 0& 1 \\ \text{slash} &0 & 1 \\ \text{antitrust} & 0 & 1 \\ \vdots & \vdots } \quad \quad \hat{\bm{\Omega}}_{\text{INFL}} =\!\!\!\! \kbordermatrix{ & & \ \\ \text{food } & 1& 0 \\ \text{customer } & 1 & 0 \\ \text{frustration } & 1& 0 \\ \text{hardware} & 1 & 0 \\ \text{vote } & 1& 0 \\ \text{involved } & 1& 0 \\ \text{disruption } & 1 & 0 \\ \text{clothes } & 1 & 0 \\ \text{suspend } & 1 & 0 \\ \text{shortage } & 0 & 1 \\ \text{gasoline } & 0 & 1 \\ \text{inflation } & 0& 1 \\ \vdots & \vdots } \]
A two-dimensional selection matrix generates the best performance in both cases. We see that most selected tokens by our optimization strategy have an intuitive relation with the target variable. This data-driven selection could be improved either (i) by augmenting the population size, the number of iterations, or the number of active tokens in the algorithm or (ii) by proceeding with a final human-supervised pruning step.
Table (ref) reports the root mean squared error (RMSE) of linear regression models applied to tracking the VIX and inflation expectations using benchmarks and optimal indices across different time windows. As expected, the optimization process leads to an optimal index that exhibits lower RMSE for both the VIX and inflation expectations during the training and validation periods. The reduction in RMSE is substantial for the VIX (49% in the training window and 39% in the validation window) and inflation expectations (23% in the training window and 27% in the validation window). More importantly, the outperformance persists on the test window (data not observed during the selection matrix optimization process), reducing RMSE by 37% for the VIX and 25% for inflation expectations relative to the benchmarks.
Further examination of the differences in performance is provided in Figures (ref) and (ref), which display the cumulative squared error difference between the benchmarks and the optimal indices. An upward-sloping curve indicates that the optimal selection matrix outperforms the benchmark. For the VIX, the outperformance of the optimal index remains relatively consistent during the test window, except for a notable flattening towards the end of the sample period, corresponding to a lower volatility period after the initial spike due to the COVID-19 pandemic. For inflation expectations, the outperformance of the optimal index is evident from 2016 to mid-2018, followed by a slight underperformance until 2020. Interestingly, the optimal index exhibits superior performance as inflation expectations rise towards the end of the sample period.
In this research, we address a critical aspect often overlooked in economic and financial analysis using textual data---the text selection process. We introduce an algorithm that determine which set of texts, among a large corpus, leads to a text-based index that is optimal for a specific objective---typically, an index that maximizes the contemporaneous relation or the predictive performance with respect to a target variable, such as inflation. Our approach, based on a genetic algorithm and tailored crossover and mutation operators, offers a data-driven and systematic text selection procedure.
We illustrate the relevance of our approach using a large collection of news articles from the Wall Street Journal. In particular, we show how to improve the existing news-based VIX index by baker2019policy or the news-based inflation expectations index by pfajfar2013news and marcellino2022demand.