Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
53,778 characters · 11 sections · 37 citation commands
blackNowcasting the euro area with social media data
\doublespacing
JEL: E31, C32, C53, C55
Keywords: Social Media, Nowcasting, Natural Language Processing, Sentiment Analysis, Big Data, Large Language Models.
Monitoring price and labor market developments is a key task in policy institutions. Since the release of hard data is usually subject to considerable delays, nowcasting models in combination with high-frequency information can be used to bridge the gap. Besides the non-trivial choice of what model to use to generate the nowcasts, practitioners have to select the set of timely information from which they can extract useful signals about the economic status quo. In this paper, we use social network data for nowcasting inflation and unemployment in the euro area.
Our work provides three key contributions: first, we use a large language model (LLM) to process social network data from Reddit to construct forward-looking and context-sensitive indicators related to inflation and unemployment in the euro area. The LLM is notably better at identifying textual information about future economic developments that are relevant for the euro area than other language processing techniques such as dictionaries. Second, we explore ways to incorporate the role of social interactions in the construction of the time series, using a variety of filters, scoring mechanisms, and decision thresholds to distinguish signals from noise in user-generated comments. We show that accounting for social interaction refines sentiment analysis and improves the robustness of economic signals. Third, we rigorously test the out-of-sample validity of the constructed indicators in a nowcasting application. Our findings reveal a consistent out-of-sample improvement in performance with respect to other state-of-the art sentiment indicators and financial variables, especially when we refine our series with social interaction. We achieve gains ranging from 13 percentage points for food price inflation to a minimum of 5 percentage points for youth unemployment.
This paper relates to two strands of research. On the one hand, the nowcasting literature has provided broad evidence that high-frequency, forward-looking variables improve performance in the euro area for important aggregates such as output, inflation, and employment. Studies have used, among others, surveys giannone2009nowcasting, banbura2023nowcasting, cascaldi2024back, newspaper articles ashwin2024nowcasting, barbaglia2024forecasting, financial market indicators breitung2015forecasting, aliaj2023nowcasting, Google Trends ferrara2023google, or price scanner data beck2024nowcasting. In spite of the ubiquity of social media and their key role in lowering the cost of information acquisition han2013social, goyal2017information, the literature has so far made sparse use of such data for the purpose of macroeconomic monitoring. Notable exceptions are angelico2022can who use X/Twitter data for inflation expectations in Italy and benatti2020high who leverage LinkedIn data to track European unemployment. Our work adds new insights to this research area by exploring the Reddit social network and evaluating how to use the social interaction arising in the community for signal construction.
On the methodological side, the literature on extracting macroeconomic information from unstructured data has recently started to evolve away from traditional natural language processing (NLP) towards increasingly sophisticated methods such as generative artificial intelligence, for example through the use of LLMs. First-generation NLP methodologies focused on structured extraction and classification of text patterns. Early contributions include baker2016measuring, who developed the Economic Policy Uncertainty (EPU) index from newspaper data, and more recent works such as aprigliano2023power, who apply topic modeling and frequency analysis to social and news data for real-time indicators, or barbaglia2024forecasting, who forecast European GDP with newspaper sentiment extracted with a dedicated economic dictionary. The advent of transformer-based models vaswani2017attention has significantly advanced the field by enabling deeper contextual understanding of more general types of documents. Large language models like LLaMa LLaMa2023 can process and generate high-quality text with minimal or no task-specific fine-tuning, offering a more scalable approach to extracting insights from unstructured textual data. As a consequence, LLMs have found increasing application in the macroeconomic literature as well. carriero2024macroeconomic apply LLMs directly to forecast US time series data, while faria2024artificial and hansen2024simulating show how prompting enables LLMs to act as forecasters, mimicking survey expectations without structured inputs. Closest to our paper, bybee2023surveying and horton2023large use LLMs to extract sentiment and simulate economic beliefs from US newspaper articles, demonstrating their ability to go beyond pattern recognition and emulate human expectations. We add to this branch of research by applying LLM-driven sentiment analysis for nowcasting macroeconomic variables in the euro area.
The remainder of the paper is structured as follows: section (ref) explains the raw data extraction and transformation into time series signals using the LLM; section (ref) presents the setup of the nowcasting horse-race and discusses the results in detail; section (ref) concludes.
We use data from Reddit, a social media platform composed of communities, known as “subreddits", dedicated to specific topics such as politics, technology, and economics. Users can share news items, opinions, and experiences related to economic trends, market behavior, and technological developments, thus producing a rich and timely dataset of user-generated content. A key difference between Redddit and traditional media outlets, is the fact that the users on Reddit make an active selection of the news items/posts they submit. While sentiment indicators that are constructed from newspaper articles reflect the supply side of news generation, Reddit users disclose their interest about selected topics though the selection of posts, thus acting as a potentially important filter of the currently available wealth of information. Moreover, users can engage with the posted content through commenting and voting, which provides an additional layer of signal processing. This combination of characteristics allows us to use Reddit as a source for gauging agents' real-time sentiment.
To manage the different levels of information that Reddit has to offer, we proceed in steps, as shown in Figure (ref). First, we select the community of interest for our study, the subreddit r/europe, which is the largest community discussing European-related news and economic developments. Second, within the selected community, we identify (1) submissions and (2) the comments attached to them. The first row of Table (ref) shows the total number of comments and submissions recorded for the time period between 2012 and 2023. Given the very large number of items, we use a keyword-filtering approach to pre-select the submissions that are relevant for the macroeconomic concepts inflation and unemployment.\footnote{Submissions should contain at least one of the following keywords to be selected: inflation-related: inflation, deflation, hyperinflation, price; unemployment-related: unemployment, employment, unemployed, job. We notice that changing the set of keywords used to filter does not significantly alter the results. Indeed, the most important selecting words turn out to be the economic concept we wish to target, i.e. inflation or unemployment.} This yields 4,825 inflation-related and 1,934 unemployment-related submissions to be analyzed.
Third, for the associated comments we make a distinction between those that immediately follow the original submission and the full collection of comments. While the direct comments are likely to be the most relevant ones for the topic raised in the submission, additional layers of comments may still contain valuable information. To make this distinction, we recover the full network structure of comments following an initial submission. We call the unfiltered set of comments that are directly attached to the submission first-level. The set of comments across all levels, on the other hand, is very large. Therefore, we make use of the keyword filtering approach discussed above to subset it again. This collection is labeled keyword. The final numbers of first level and keyword-filtered comments are summarized in Table (ref). In addition, each comment and submission can be upvoted and downvoted by the users and we record the net scores for all posts as the difference between upvotes and downvotes. Following these steps, the selected submissions and comments are fed to the LLM, to extract a signal that we use to nowcast inflation and unemployment. We explain this step in more detail below.
Due to their generality, LLMs are particularly suited for handling the challenges posed by the linguistic characteristics of Reddit communities, as noted in long2023just. Unlike the language found on conventional news outlets, Reddit users often write in community-specific slang, use abbreviations, and generally informal language that strongly deviates from standard dictionaries commonly found in economic research loughran2011liability. By using an LLM, we can capture the nuanced, community-specific signals present in Reddit data without the need of extensive fine-tuning or adjustment of existing dictionaries. We use the model LLaMa-3-70b-instruct on local servers of the European Commission. For the present research, we instruct the model to classify each submission and comment into one of three categories (UP, DOWN, NEUTRAL) based on the forward-looking signal contained in each of the provided items. The exact prompt is reported below:
\captionsetup{type=figure} \captionof{figure}{LLM prompt used with LLaMa-3-70b-instruct. concept is either unemployment rate or inflation rate, df['title'][i] contains the submission or comment.} {15pt}
As a benchmark, we adopt two dictionary-based approaches to compute the three-outcome signal taking into account positive and negative words appearing in each Reddit submission. In Appendix (ref) we provide the dictionary, that is based on the work of granziera2025speaking. The second dictionary of consoli2022fine is substantially more comprehensive and therefore not fully reported in the Appendix. Both works have been shown to be successful in extracting meaningful signals from economic texts such as Fed speeches granziera2025speaking and newspapers barbaglia2024forecasting.
A key benefit of using the LLM over other methods lies in its generality. To quantify the gains over simple dictionary approaches and simultaneously assess the replicability of the result, we calculate the accuracy of the LLM in terms of the F1 score. Since our classification yields three possible outcomes for each piece of text, the F1 score is computed as the unweighted average of the F1 score for each of the three categories. The F1 score per category is computed as
where TP is the category's true positive rate, FP is the false positive rate and FN is the false negative rate. Evidently, there is no objective ground truth for the sentiment attached to Reddit submissions or comments that could be used to compute and assess error statistics for the LLM. Instead, we have to rely on a human-labeling approach to gauge the accuracy of the models. We randomly select a subset of 483 inflation and 194 unemployment related submissions (10% of the total number of submissions for the two targets) which we manually label using the UP, DOWN, NEUTRAL scale. We ensure that the time coverage over the selected submissions is even and not bunched in a few specific years. We then compute the F1 scores for the LLM and the two dictionaries explained above. To ensure robustness, we vary the temperature parameter of the LLM on a grid between 0.1 and 0.9 in steps of 0.2. This nonnegative parameter controls the “creativity" of LLM responses. Higher values -- the maximum being 1 -- typically produce more volatile responses and 0 produces deterministic answers. For each temperature, the LLM is asked to score the same 483/194 submissions 100 times to obtain a measure of variability for each configuration. The dictionary approach of granziera2025speaking achieves a score of 0.340 for inflation and 0.333 for unemployment. The consoli2022fine dictionary approach reaches 0.290 for inflation and 0.339 for unemployment.
The results for the LLM are visualized in Figure (ref), which shows that it far outperforms the scores of the dictionary approaches for both variables. Unemployment scores are slightly higher than those for inflation. Overall, the scores lie around 0.71 (inflation) and 0.75 (unemployment). In nearly all cases, we can see that increasing the temperature widens the range of possible accuracy scores, while the median stays essentially stable. We have used a temperature of 0.5 throughout the following analysis.
Building on the classifications obtained via the LLM, we construct the time series indicators $X_t$ for submissions only and $\bar{X}_t$ for signals, which combine both submissions and comments. Let us start with the submissions-only signal. To create this, we aggregate all signals $S_i \in \left\{\text{UP, DOWN, NEUTRAL}\right\}$ for submissions posted on day $t$, computing the daily sum, where UP takes value 1, DOWN takes value -1, and NEUTRAL is treated as 0:
$N_{t}$ represents the total number of submissions recorded on a given day. If no submissions are available in $t$, the indicator is assigned a value of zero. $X_t$ is then smoothed using a backward looking moving average (MA) filter of varying window sizes. The effect of the smoothing parameter on nowcasting performance is discussed in Appendix (ref).
Next, we turn to the construction of time series signals $\bar{X}_t$, which also incorporate social interaction, a key novelty of this paper. The steps are illustrated in Figure (ref).
To do this, for each submission $i$ we compute a score ${L}_{i} \in [-1,1] $, which combines not only the submission's original signal ($S_{i}$), but also the signals derived from the $J$ comments $C_{i,j} \in \left\{\text{UP, DOWN, NEUTRAL}\right\}$ associated to it. If a submission receives no comments, the original signal remains unchanged.\footnote{A valid concern is that comments could arrive much later than the submission, invalidating their use for a nowcasting or forecasting application. We checked the timing of all the comments and found that they are posted at most one week after the main submission. This ensures a significant reduction of any possible look-ahead bias coming from comments arriving at later points in time. To corroborate this, the weekly information flow is discussed in Appendix (ref).}
Comments $C_{i,j}$ can either be those that are directly attached to the submission, but unfiltered by keywords (first level), or the set of all keyword-filtered comments (keyword), as explained in section (ref). We treat each comment as a vote. If the LLM assigns UP to the comment, the vote is counted as 1. If it assigns DOWN, the vote is -1 and 0 for NEUTRAL classifications. As noted in section (ref), comments can also receive upvotes and downvotes by users. The net score of these can be used as a weight on the vote that each comment casts. Taking the average of the (possibly weighted) votes of all comments in addition to that of the original submission yields the score ${L}_{i}$ in ((ref)). Finally, we need a decision rule to transform the score $L_i$ back to the UP, DOWN, NEUTRAL scale. We achieve this through the threshold parameter $\tau$. This parameter is varied on a discrete grid, to evaluate its impact on the quality of the signal. Based on the voting outcome, the new label $\bar{S}_i$ for submission $i$ is obtained:
Once this process is finished and all new submission labels $\bar{S}_i$ are obtained, the daily scores are re-computed as $\bar{X}_{t}$ according to:
The resulting signals $\bar{X}_{t}$ are also smoothed with an MA filter of different window lengths.
Crucially, this voting scheme allows users to overturn the originally assigned score of the submission $S_i$, if they disagree with the original sentiment. Therefore, social interaction acts as regularization device, which introduces checks-and-balances into the signal construction. An example of such a re-classification is presented in Appendix (ref). In sum, the time series involving not only submissions, but also social interaction through comments differ along four dimensions: (1) unfiltered first level or keyword-filtered comments, (2) usage or not of up- and downvotes, (3) thresholds used for voting outcomes, and (4) MA smoothing windows. All of these choices are evaluated out-of-sample and results are reported in Appendix (ref). At the end we have a total of 120 series for each of the two target variables, inflation and unemployment. Examples of the daily signals from Reddit for inflation and unemployment are reported in Figure (ref). Our Reddit signals track HICP inflation and the low frequency component of the unemployment rate remarkably well. Therefore, we proceed by evaluating the informativeness of the signal in an out-of-sample application next.
To test whether our Reddit signals are useful for real-time monitoring of macroeconomic conditions in the Euro area, we set up a nowcasting application. The target variables are year-on-year HICP inflation and its components for energy, services, food and core, as well as the overall unemployment rate plus those for people over and under the age of 25 years in the euro area. Target variables are obtained from the EA-MD data base created by barigozzi2024ea, with the exception of food price inflation, which comes from the European Commission. These are useful nowcasting targets, as inflation data are released in the middle of the month after the reference period (a flash estimate becomes available around the end of the reference period/the beginning of the following month) and unemployment rate data are typically released one month after the end of the reference period. Moreover, past research has shown that price and labor market developments are among the topics about which people care most deeply (see section IV in barsky2012information). Engagement around these topics on social media may therefore plausibly contain early signals of ongoing macroeconomic dynamics. Since Reddit users tend to be relatively young non-experts, we would expect that opinions related to readily observed indicators such as food prices, energy prices or youth unemployment would provide better results relative to those related to broader inflation and unemployment measures (see also d2021exposure). The out-of-sample period ranges from January 2018 to December 2023, leaving an initial estimation sample from January 2012 to December 2017, which we then expand recursively.
Our set of candidate indicators includes all 120 Reddit indicators whose construction is detailed in section (ref), taken one by one. To this set of social media variables we add for comparison the daily newspaper sentiment indices computed by barbaglia2024forecasting, inflation swaps for 1 to 5 years and the Brent oil price (FRED mnemonic DCOILBRENTEU), reflecting the fact that commodity prices have been shown to aid forecasting performance for inflation breitung2015forecasting. The newspaper sentiment indicators of barbaglia2024forecasting are available for Germany, France, Italy, and Germany. To obtain an aggregate index per concept, we take the first principal components of the available data. We have considered indicators for inflation, unemployment and monetary policy sentiment.
We use MIDAS regressions to combine monthly data with daily data. In particular, we estimate the following MIDAS-AR regression ghysels2016mixed:
where we choose standard second-degree Almon polynomials to shrink the parameters for the weights:
For each target-predictor combination, the lag order $q$ in $\beta(L)$ for the daily data is determined based on the Akaike information criterion (AIC) each time a new nowcast is produced. The lag order $p$ in $\alpha(L)$ for the endogenous data is fixed at 1. We choose this simple setup because the emphasis of our exercise is on comparing information sets rather than models. Moreover, we note that the exercise is not fully real-time, as we do not use data vintages. However, price indexes and unemployment rates are hardly subject to revisions. For Reddit data (and our other daily indicators) revisions are also negligible and arise only if certain submissions or comments are removed by the moderators or if users delete them themselves. Additionally, as our focus is on assessing the information content of individual Reddit signals and the role social interaction plays, we have not explored the potential benefits of including more than a single Reddit indicator at a time in the nowcast. This could be implemented using variable selection techniques as in babii2022machine. We leave this extension for future research.
The information setting of the nowcasting evaluation is as follows: in a given month, we consider the monthly inflation and unemployment data up to the previous month to be fully observed by the nowcaster. In addition, the nowcaster gets access to daily data through the entire current month. For example, in February 2025, the nowcaster wants to obtain predictions for inflation and unemployment. She has access to monthly observations for both variables through January 2025 and to daily signals all through February 2025. In the case of inflation this reflects the availability of the flash estimate, which comes out between the end of January 2025 and the beginning of February 2025. For the case of unemployment data, our approach marks a conservative benchmark, as we are providing data that is technically released a few weeks later, thus reducing the comparative edge of daily data.
We evaluate the usefulness of the Reddit signals by comparing the root mean squared forecast errors (RMSFE), the mean absolute forecast errors (MAFE) and the continuously ranked probability score (CRPS) of the MIDAS regressions with those of a nowcast based on a monthly AR(1) model, which serves as a benchmark. For the density forecasts we assume normal errors and simulate 1000 nowcasts per model for each period in the evaluation sample. We assess the difference in nowcasting performances for all three error metrics using the Diebold-Mariano test in the version of harvey1997testing. While our models are nested, we rely on the arguments made in clark2015macroeconomic that this constitutes a conservative approach to significance testing. Moreover, since the models are nested we use the p-values of one-sided tests in which the AR(1) errors are used as the baseline and the MIDAS nowcast errors as the alternative.
We begin by presenting the results for the eight target variables for the best-performing daily Reddit specification, and we compare them with the best performers among sentiment indicators, financial indicators (swaps), and the oil price.
Table (ref) shows that the Reddit indicator generally outperforms the other variables in terms of point and density accuracy. The nowcasting gains using Reddit range from 13 percentage points (food price inflation) to at least 5 percentage points (youth unemployment) relative to the next best specification. It significantly beats the AR(1) benchmark in most cases and presents particularly strong performance gains for food price inflation and the overall HICP index. Gains for unemployment nowcasting are sizable but statistically significant only for the MAFE. Somewhat surprisingly, the nowcast improvements are slightly larger for the unemployment rate for people over the age of 25. Inflation-related newspaper sentiment and swaps also improve frequently over the AR(1), but to a smaller extent than the Reddit indicator. The sentiment index for unemployment proves a solid competitor as well. Models involving oil prices are on par with the AR(1). In all cases, the specification involving not only Reddit submissions, but also comments achieves the best results.
Figure (ref) shows the evolution of the forecasting performance of the different MIDAS specifications against the AR(1) over time. In terms of HICP inflation, we see that the inflation swaps performed better than the Reddit indicators after the onset of the COVID-19 recession and through the first part of the high inflation period between 2021 and 2023. As the inflation rate starts to drop, this picture reverses and the performance of the Reddit indicator improves markedly. This suggests that the sentiment among social media users reversed faster than on financial markets. For the case of the unemployment rate, we see a single significant improvement in the nowcasting performance after the COVID-19 recession for both the sentiment indicator and the Reddit signal. Otherwise performances are quite stable and the results seem to be driven by a few large shocks that were missed badly by the AR(1). These results appear in line with the literature which has argued that high frequency information is particularly useful in times of turmoil, but not so much in tranquil times barbaglia2023testing. We confirm that gains are especially visible during the COVID19 period and the subsequent high inflation episode by means of a Giacomini-Rossi test, reported in Appendix (ref). \\
A key take-away from the horse-race is the following: monitoring social media activity around the topics of inflation and unemployment produces a clearer picture of ongoing price and employment dynamics in the euro area. This is particularly true if the data are collected at daily frequency and take into consideration the comments of other users. In this section we aim to shed some additional light on the way social interaction improves the social media signal. When we compare the classification of all posts based on the LLM using only the original piece of information (submission) and the revised classification which takes into account comments, we observe that in the case of inflation, the submission classification is re-classified in 11% of the cases. For unemployment the share increases to 15%. In both cases, social interaction leads to hundreds of re-classifications. Since nothing else changes between the MIDAS model that uses only submissions and the one that accounts for social interaction, it is this re-classification which reduces nowcasting errors.
Tables (ref) and (ref) show the direction of change that comes about through the incorporation of comments into the signal for the winning specifications. In the case of inflation, the total signal is revised upwards whenever the re-classification is from -1 to 0, -1 to 1 and from 0 to 1. This happens 366 times (counting -1 to 1 twice) while only 171 downgrades occur, a ratio of 2.1. Therefore, the overall sentiment for the direction of inflation in the sample we have considered was generally revised upwards. For unemployment, there are 188 upward revisions and 105 downward revisions, a ratio of 1.8. The results of the horse race suggest that this sharpens the signal, as the nowcast accuracy improves. Since we have observed the clearest nowcasting gains in unusual time periods, we also check if these re-classifications are concentrated in and around these times. Figure (ref) shows that this is not the case, which leads us to conclude that this type of regularization is not episodic but constitutes a systematic feature of social media signals that researchers can use to improve their information set.
This paper contributes three novelties to the literature. First, by employing a state-of-the-art LLM, we extract forward-looking and context-sensitive signals related to inflation and unemployment in the euro area from millions of Reddit submissions and comments. Relying on this specific data source represents a valuable filter, because users make an active selection of the posts they deem important enough for submission to the network. Moreover, we show that in the context of analyzing diverse economic topics on social media, using an LLM is more accurate than other popular language processing approaches such as dictionaries.
Second, we suggest a method to include social interaction among users in the construction of the signals by means of a voting scheme. The sentiment conveyed in the discussions triggered by the initial submissions to the r/europe thread can be used to refine the sentiment scores if users disagree with the message about future economic developments contained in the submission alone. We explore different facets of constructing these signals, based on the depth of the comment network, weighting through up- and downvotes, and decision thresholds.
Third, we perform a rigorous out-of-sample evaluation of the signal quality of all constructed social media indicators. Our empirical results show consistent gains in out-of-sample nowcasting accuracy when using the signals which incorporate social interaction, even when compared to daily newspaper sentiment and financial variables. Improvements range from up to 13 percentage points for food price inflation to at least 5 percentage points for youth unemployment, highlighting the added predictive value of social media-derived insights.
We conclude that the application of AI tools to the analysis of the vast, multi-faceted information contained in social media, specifically Reddit, constitutes a useful addition to the toolkit available to economic forecasters and nowcasters. We view as promising areas for future research the use of more sophisticated, possibly multivariate nowcasting models, which allow for the inclusion of more than one Reddit indicator and competitor variables such as Bayesian VARs, Dynamic Factor Models or variable selection methods. Moreover, an extension of the analysis to sub-European Reddit threads may be beneficial for obtaining timely information on economic developments tailored to specific countries.