Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
92,518 characters · 30 sections · 83 citation commands
Macroeconomic Forecasting with Large Language Models
\thispagestyle{empty}
\doublespacing
The recent emergence of Large Language Models ({LLM}\xspaces) has reshaped the landscape of natural language processing, ushering in a new era of computational linguistics. Bolstered by advancements in machine learning and deep neural networks, {LLM}\xspaces have garnered widespread attention for their remarkable ability to understand and generate human-like text. This transformative technology has revolutionized various applications, ranging from machine translation and sentiment analysis to chatbots and content generation. By leveraging vast amounts of text data and sophisticated algorithms, {LLM}\xspaces have demonstrated unparalleled proficiency in capturing linguistic nuances, contextual dependencies, and semantic meanings.\footnote{For example, the first draft of the abstract of this paper was generated with a single prompt by chatGPT 3.5.} As a result, they are increasingly becoming invaluable tools for researchers, developers, and businesses.
In this paper we focus on an even more recent development, which is the use of {LLM}\xspaces to forecast time series data. While {LLM}\xspaces have predominantly been associated with natural language processing tasks, their versatility and adaptability have sparked interest in exploring their capabilities beyond linguistic domains. By leveraging the computational power and flexibility inherent in machine learning algorithms, {LLM}\xspaces promise to uncover intricate nonlinear relationships, capturing latent dynamics, and adapting to evolving data patterns.
This latest iteration of {LLM}\xspaces, trained specifically with the goal of forecasting time series data, is referred to as either Time Series Language Models ({TSLM}\xspaces) or Time Series Foundational Models ({{TSFM}}\xspaces) in the Machine Learning and Artificial Intelligence literature.\footnote{In this paper, we prefer to use the {TSLM}\xspace acronym, to make it explicit that these models are derived from textual {LLM}\xspaces and as such share many similarities with them both in terms of architecture and model design.} Several {TSLM}\xspaces have already been produced and are publicly available, including IBM's Tiny Time Mixers ekambaram2024ttms, {Time-LLM}\xspace jin2023time, { LagLlama}\xspace rasul2023lag, Google's {TimesFM}\xspace das2024decoder Salesforce's {Moirai}\xspace woo2024unified, and Nixtla's {Time-GPT}\xspace garza2023timegpt1. All these contributions show that {TSLM}\xspaces can produce gains in forecast accuracy, but in their empirical evaluations they tend to focus on a variety of datasets from very different domains and do not offer any specific information on how these models would fair in forecasting macroeconomic variables specifically.
Our first contribution in this paper is a thorough investigation on how {TSLM}\xspaces perform in predicting macroeconomic time series. A second contribution is a detailed comparison of their performance versus state-of-the-art time series methods such as Bayesian Vector Autoregressions (BVARs) and Factor Models, as well as Neural Networks and Bayesian Additive Regression Trees (BART), which have both become very popular nonlinear models for macroeconomic forecasting. A third contribution is the development of a novel hybrid approach that combines the structural rigor of traditional econometric models with the adaptability of {TSLM}\xspaces, leading to consistent improvements in forecast accuracy across variables and horizons. In these endeavors, we focus on forecasting the variables contained in the FRED-MD dataset, a comprehensive repository of more than one hundred monthly macroeconomic variables curated by the Federal Reserve Economic Data (FRED) system.\footnote{See \href{https://research.stlouisfed.org/econ/mccracken/fred-databases/}{https://research.stlouisfed.org/econ/mccracken/fred-databases/}.} This dataset encompasses a diverse array of economic indicators, and it is considered a benchmark in macroeconomic forecasting. A non-exhaustive lists of contributions using this data set to study the performance of alternative models in macroeconomic forecasting includes Stock:Watson:2006, Banburaetal2010, Carriero:etal:JBES:2016, CARRIERO2019JOE, KOROBILIS2019241, KOOP2019135, chan2022asymmetric.
As noted in FL2024, there are some characteristics of LLM models that makes it problematic to use them in the pseudo out-of-sample forecasting exercise that is typically performed in the empirical macroeconomics literature. One major challenge is that LLMs are pretrained by developers on datasets of their choice, giving the researcher little to no control over the training data. For example, of the LLMs considered in this work, three out of five of them (including the best performing one, Salesforce's {Moirai}\xspace) list in their training data a large subset of the series we set out to forecast in this paper. To further complicate matters, since the training data lacks timestamps it's hard to retrain the model up to a certain date, making it difficult to reproduce the type of real-time analysis that an economic forecaster is mostly concerned with. Solving these issues by retraining LLMs on subsets of data and adding data to the training set is usually not possible due to high hardware requirements. But having said that, it is important to recognize that even including the dataset of interest in real time would not remove the issue of a contaminated training set, i.e. including information that the forecaster would not have access to in real-time.
More specifically, our empirical application is focused on forecasting all variables contained in the FRED monthly database using data ranging from 1960 to 2022. The picture emerging from our analysis is one in which only two of the five {TSLM}\xspaces we consider are competitive against a simple AR benchmark (Salesforce's {Moirai}\xspace and Google's {TimesFM}\xspace). Moreover, when these more competitive {TSLM}\xspaces are stacked against Bayesian VARs and Factor Models, their forecasting performance is broadly comparable, if not slightly inferior. In particular, we find that the forecasting gains achieved by the econometric models tend to be more stable while {TSLM}\xspaces can perform very well for a handful of series but also show less reliability at times, as they seem to be more prone to generating the occasional unreasonable forecast.\footnote{This result may not appear to some as totally surprising. When working with textual LLMs, we are sometimes exposed to “Model hallucinations”, i.e. the tendency of the model to generate content that is irrelevant, made-up, or inconsistent with the input data.} On the other hand, {TSLM}\xspaces seem to work relatively better in the post-Covid-19 era (with the important caveat that the training set of these models does include information from the pandemic and post-pandemic period). While these results are based on using the {TSLM}\xspace models in a zero-shot forecasting manner, we also find that fine-tuning the {TSLM}\xspace models does not yield to significant improvements in their forecast accuracy.\footnote{By “zero-shot”, we mean that the model is first trained on a large amount of data, and is then directly applied to solve a new task without re-training, or fine-tuning, its parameters on the new dataset.}
There are a few other papers that are looking at LLM in the macroeconomics and finance literature, and it is reasonable to expect that many more will follow. For example bybee2023surveying feed into an LLM model, OpenAI’s GPT-3.5, a historical sample of news articles from The Wall Street Journal (WSJ) and ask the model to predict various financial and macroeconomic quantities, which they then aggregate into a time-series of monthly and quarterly expectations that are compared to a variety of existing survey forecasts. Chen_Kelly_Xiu_2022 feed global news text data from Thomson Reuters Real-time News Feed (RTRS) and Third Party Archive (3PTY) in 13 different languages into pretrained LLMs, namely BERT (developed by Google), RoBERTa (by Meta), and OPT (by Meta), and use the resulting sentiment scores to predict firm-level daily returns. Kim_et_al_WP_2024 feed standardized and anonymous financial statements to GPT4 and instruct the model to analyze them to determine the direction of future earnings. They find that LLMs outperforms financial analysts in their ability to predict earnings changes. These approaches focus on leveraging the typical LLMs abilities in natural language processing, while this paper is focused on the recent efforts of training LLMs specifically for time-series forecasting. Zhou_et_al_WP_2025 study whether both DeepSeek and ChatGPT can be used to predict the stock market and find that a good news ratio extracted from prompting ChatGPT is positively correlated with contemporaneous market returns and significantly predicts subsequent returns up to six months out. The closest work we are aware of in the macroeconomics literature is FL2024, which looks at time series LLMs, but only focuses on one LLM and one target variable (inflation). In contrast to this, in this paper we take on the ambitious goal of looking at the performance of a variety of {TSLM}\xspaces, comparing their accuracy against state of the art macroeconomic forecasting approaches, across a large set of macroeconomic indicators.
The paper is organized as follows. (ref) introduces LLMs and {TSLM}\xspaces and provides a discussion on how they are being leveraged to perform time series forecasting. (ref) describes the econometric models we consider as state of the art benchmarks in our exercise. (ref) describes the dataset and the design of the out-of-sample pseudo real time forecasting exercise and (ref) and (ref) present our empirical results. (ref) describes a novel hybrid approach that integrates econometric models with the flexibility and nonlinearity of {TSLM}\xspaces. (ref) provides some concluding remarks and directions for further research. An Appendix provides technical details on the models considered, some additional results, and details on the data transformations used in the empirical analysis.
In recent years, Foundational Models (FMs), i.e. very large neural network models with billions of parameters have received tremendous attention in computer science fields due to their ability to capture complex relationships within massive datasets. One of the most astonishing features of FMs is their flexibility, i.e. their applicability to various tasks across many different domains in a zero-shot manner, effectively eliminating the need to train specialized models for each separate task. Within this context, textual LLMs have emerged as a new way of understanding language, generating text and images, and conversing in natural language, and have completely revolutionized the way we interact with technology achiam2023gpt. State-of-the-art textual LLMs and Chatbots, such as Open AI's ChatGPT and Meta's Llama, are now being used extensively across various natural language processing tasks, such as Information Retrieval and Scientific Research, but also Code Generation and Debugging.
Inspired by these successes, researchers are now exploring the usefulness of LLMs in other settings. Along these lines, Lu_Grover_Abbeel_Mordatch_2022 find that LLMs trained exclusively on textual data can effectively solve sequence modeling tasks in other modalities, such as numerical computation, vision, and protein fold prediction. Among these extensions, we are witnessing the emergence of Time Series Language Models ({TSLM}\xspace), bridging the gap between LLMs original text data training and the numerical nature of time series data. These models are sometimes referred to in the literature as Time Series Foundational Models ({{TSFM}}\xspace).
TSLMs are now being used to accomplish a variety of time series related tasks, ranging from prediction and classification to anomaly detection and data imputation. Notable examples of this new wave of model development include rasul2023lag, goswami2024moment, ekambaram2024ttms, garza2023timegpt1, das2024decoder, and ansari2024chronos. At the core of this very recent advancements is a simple but fundamental point. While there are in principle significant differences between a textual LLM, whose main task is essentially to predict the next word, and a time series forecasting model, whose main objective is to predict the next value of one or multiple time series jointly, inherently these two tasks are deeply intertwined. They both aim at learning the sequential structure of the data from the past and present history of some features and with that attempt to predict a sequence of future outcomes.
In the case of a {TSLM}\xspace, the pretraining is done on a possibly very large set of $N$ time series observed over the time periods $t=1,\dots,T$ which we collect in $\bm{X}_{1:T}=\left(x_{1,1:T},...,x_{N,1:T}\right)$. Pretraining yields a very large set of coefficients $\widehat{\theta} \in \Theta \subseteq \mathbb{R}^d$ which describe a forecasting function $f_{\widehat{\theta}}$. Then, given $f_{\widehat{\theta}}$ and the current and past values of a time series of interest $y_{1:T}$ (not necessarily in the training set), the {TSLM}\xspace attempts to predict its future $h$ values $y_{T+1:T+h}$ using\footnote{To keep the notation simple, we have assumed here a balanced set of features, i.e. all available over the same time period. If that is not the case, then we could instead write this as $\bm{X}_{1:T}=\left(x_{1,\underline{t}_1:T_1},...,x_{N,\underline{t}_N:T_N}\right)$, where $\underline{t}_i$ and $T_i$ would denote, respectively, the first and last observation of feature $i$, $i \in \left\{1,...,N\right\}$.}
The main difference with LLMs is that while natural language consists of words from a finite vocabulary, time series are real-valued.
The earliest applications of LLMs to time series forecasting were built on simply prompting existing textual LLMs with time series data. For example, PromptCast Xue_etal_2023 relies on converting numerical time series into text prompts and forecasting in a sentence-to-sentence manner.\footnote{Here is a weather forecasting example, to demonstrate how this approach would work: “From $t=1$ to $t=T$, the average temperature of region $i$ was $X_i$ degree on each day. What is the temperature going to be on day $T+h$?”} There are multiple reasons why these approaches are sub-optimal. For once, these models requires a variable specific template to ask forecasting questions. In addition, using textual LLMs as is may lead to poor forecasting performances, mainly due to lack of real-world time series data in the training of these LLMs in the first place.
The most recent versions of {TSLM}\xspaces have evolved from this initial approach and now build on quantization-based methods, i.e. a process through which the numerical data is converted into discrete representations first, before becoming an input to the training of the LLM. We now turn to describing the basic building blocks of a {TSLM}\xspace, drawing whenever possible parallels to their textual counterparts for clarity. The specific choices and configurations of these building blocks define the unique characteristics of the various models. An overview of {TSLM}\xspace architecture is shown in (ref).
In textual LLMs e.g. ChatGPT, the process of text understanding begins with tokenization, which breaks down sentences into their fundamental semantic units -- tokens. Tokens can be individual words, characters, or subwords. Tokens allow LLMs to identify and learn the relationships between semantic units, leading to a better text understanding. As we discussed above, time series data lack inherent well-defined semantic units such as words. To circumvent this issue, {TSLM}\xspaces rely on scaling, patching, and quantization techniques to transform numerical sequence data to a format "token" suitable for learning patterns and trends within data.\footnote{It is worth noting here that a lack of fundamental semantic units presents also an opportunity to apply domain knowledge to derive better tokens, and thus potentially shepherding field of {TSLM}\xspaces for macroeconomic forecasting.} The common schemes utilized by current {TSLM}\xspaces are listed below. Note that not all {TSLM}\xspaces implement each scheme; they choose what works for their training process.\looseness=-1
Patching
Patching is a process used to divide time series into shorter, fixed-length segments or windows called patches. These patches can be treated as tokens, capturing local patterns. For example, {Moirai}\xspace, {TimesFM}\xspace, {TTM}\xspace use patching to create tokens, while { LagLlama}\xspace augments each patch with lag features and time-based features to construct tokens. Importantly, the patch size is itself a hyper-parameter that can be adjusted based on the specific characteristics of the time series data. For example, larger patch sizes are more suitable for high-frequency time series, while smaller patch sizes tend to be more suitable for low frequency time series. Consider as an example a time series $x_{1:T}=\{4.7, 4.76, 6.8, 7.2, 6.1\}$. Tokenization with patching size $=3$ and overlap $=2$, will yield following tokens $\{4.7, 4.76, 6.8\}$, $\{4.76, 6.8, 7.2\}$, $\{6.8, 7.2, 6.1\}$. Without overlap the tokens will be $\{4.7, 4.76, 6.8\}$, $\{7.2, 6.1\}$.
\noindentQuantization
Quantization is used to convert numerical time series values into a set of discrete tokens, similar to finite words in textual LLMs. This is achieved by dividing the value range of each time series, or a patch or set of patches (depending on the {TSLM}\xspace) into a predefined number of bins $\mathbb{B}$. Each data point is then assigned a token, a number between $\{1, \cdots \mathbb{B}\}$ based on the bin it falls into. There are different approaches to binning - uniform quantization divides the value range into bins of equal size, and data-dependent quantization adjusts the bin sizes based on the data distribution. {Chronos}\xspace ansari2024chronos, for example, uses uniform quantization to encode time series into tokens. To continue with the previous example, let's apply uniform quantization with $\mathbb{B} = 4$ to the time series $x_{1:T}$. The mapping from $x_{1:T}$ to the four bins is carried out as follows:
Then the tokenized sequence is given by $q_{1:T}=\{1, 1, 3, 4, 3\}$.
Scaling\footnote{Scaling can also be referred to as instance normalization or standardization.}
When working with multiple time series at once, even when those are within the same dataset, it is common to have variables expressed with different scales. For example, in the case of macroeconomic variables we often see interest rates expressed in percentages, GDP growth reported in annualized growth rates, and unemployment rates displayed in absolute numbers. To ensure consistent processing, data points are typically scaled. Scaling also helps in optimization/learning for deep learning models, as varying scales may distort gradient computation. Scaling a value $x_t$ can be done as follows,
where $\tilde{x}_t$ is the scaled value. Different choices for $M$ and $S$ lead to the types of scaling used by the various {TSLM}\xspaces. For example, { LagLlama}\xspace uses $M=$ median of the context window and $S=$ inter-quartile range within the context window.\footnote{A context window is the length of a time series considered as history for forecasting.} It is also worth noting that scaling can be applied at the global level (using the value range of the entire time series), at a context window level, or at a patch level.
Embeddings
In {TSLM}\xspaces, tokenization --- through scaling, patching, or quantization --- converts raw sequence data into discrete tokens. This process is similar to how textual LLMs tokenize sentences into words or subwords. The time series tokens are then transformed into embeddings, which are dense numerical vectors. Textual embeddings capture semantic relationships between words, while in {TSLM}\xspace embeddings encode local temporal patterns such as trend or seasonality. This transformation occurs via an embedding layer, a learnable component of the transformer architecture (we discuss the architecture in the next subsection), which maps tokens to a vector space suitable for processing by attention mechanisms. While architecturally part of the transformer, the embedding process creates inputs capturing the contextual information of the time series data for the transformers.
Similar to textual LLMs, the transformer vaswani2017attention is the core model architecture used in recent {TSLM}\xspaces.\footnote{Additionally, there are non-tranformer-based architectures such as multi-layer perceptrons (MLP) used in some {TSLM}\xspaces. Of the various models we considered, all of them use a transformer architecture, except for TTM which uses a traditional MLP-based architecture. } Transformer-based models leverage self-attention mechanisms to capture long-range dependencies within time series, offering significant advantages in handling sequential data. Self-attention refers to the ability of transformers to focus on relevant parts of the time series data. In a transformer model, each token is projected into a higher dimensional space, allowing the model to capture richer information within that segment. Then, the self-attention layer applies linear transformations and uses the dot product between transformed vectors to identify relevant parts of the time series for each token. This helps in identifying long-range dependencies and relationships, agnostic of the token positions in the input time series. The original transformer architecture utilized two components, namely (i) an encoder, responsible for extracting the relevant information from the input time series and projecting/embedding it into a vector representation and (ii) a decoder, responsible for autoregressive prediction of next token conditioned on observed input sequence of tokens, and encoder output. Recently, several transformer architectures have evolved, notably encoder-decoder, decoder-only, and encoder-only. In the encoder-decoder setup, the encoder maps an input sentence of some language to a continuous representation, and the decoder generates the translation token-by-token using the input representation and previously decoded tokens. In the decoder-only case das2024decoder, the model only attends to tokens up to the current token, while in the encoder-only case woo2024unified, the model uses information from the entire sequence to generate tokens. Note that encoder-decoder and decoder-only generate output tokens in an autoregressive manner in contrast to encoder-only architecture.
Textual LLMs train on massive text data readily available from books, articles, code, and web crawls. This abundance allows them to learn underlying patterns and relationships effectively. Similarly, {TSLM}\xspace models are intended for plug-and-play forecasting, which requires large-scale time series data with diverse patterns for training. However, time series data are inherently scarce. Augmentation helps mitigate the scarcity and variability of time series datasets by generating more diverse training data. There are several approaches to time series data augmentation (see iglesias2023data for a detailed survey). These techniques typically involve convex combinations of existing sequences, combinations of ARMA processes, seasonal patterns, trends, and step functions, and combining the frequency spectrum of sequences and then converting it back to the time domain.
Pretraining refers to the process of feeding to the {TSLM}\xspaces a large amount of time series data so that the model can learn general temporal patterns and relationships and with that perform well across a wide spectrum of tasks. The time series data fed into the {TSLM}\xspace can be from various domains and span different frequencies and, as previously discussed, each time series will first be scaled and tokenized before any training occurs. Once the pretraining phase is completed, the {TSLM}\xspaces will have built a foundational knowledge base from a very general corpus of time series data, and with that can be used to perform zero-shot forecasting in a new domain.
Pretraining relies on self-supervised techniques that create tasks for the model so that it can learn directly from the input data without requiring human labeled examples. The success rate in these tasks can be measured in various ways, such as minimizing MSE or accurately predicting the quantized bins or the quantiles of the training data. As a concrete example, { LagLlama}\xspace pretraining looks for the model parameters that maximize the likelihood of the observed future values, under the assumption that these follow a Student-t distribution.
Fine-tuning is a training process that is used to adapt a pretrained {TSLM}\xspace, containing general foundational knowledge, to a specific task such as forecasting, or classification. Fine-tuning updates existing knowledge (parameters) of {TSLM}\xspace to better handle unique characteristics of the task-specific data. This can potentially improve the performance of a {TSLM}\xspace compared to zero-shot predictions for the given task. Fine-tuning can also be used to continually learn the model parameters and incorporate fresh data, improving {{TSLM}\xspace's forecasting accuracy over time. For example, in macroeconomic forecasting, we can use fine-tuning to incorporate the data from the current quarter to update the parameters of the {TSLM}\xspace.
Before moving on to describe the existing {TSLM}\xspaces, it is worth briefly discussing the difference between Deep Learning ({{DL}}\xspace) models (see for example torres2021deep, lim2021time and wen2022transformers) and {TSLM}\xspaces. {{DL}}\xspace models share many similarities with traditional time-series methods, requiring problem specific training data to learn model parameters and make forecasts. However, they differ the traditional methods in a number of ways -- using non-linear architectures, large numbers of learnable parameters, and requiring much larger training data than traditional time series methods. {TSLM}\xspaces, on the other hand, are pretrained on large number of time series datasets. As with FMs, this pretraining allows them to perform “zero-shot” forecasting. This plug-and-play approach makes {TSLM}\xspaces particularly useful and powerful. Furthermore, pretraining allows {TSLM}\xspaces to require much less domain-specific knowledge data compared to deep learning models, and if needed can be fine-tuned on specific datasets.
We now briefly describe the {TSLM}\xspace models considered in this paper.\footnote{In addition to the models listed in this section, we have also experimented with {MOMENT}\xspacegoswami2024moment, a family of time series transformer models that is pretrained on masked time series prediction task using patching. However this model is still in its very early stages and we have found its performance to be lagging significantly below the other {TSLM}\xspaces we have considered. We therefore did not include it in the comparison below.}
\paragraph{ { LagLlama}\xspace} rasul2023lag { LagLlama}\xspace is a probabilistic foundation model that extends the Llama 2 architecture touvron2023llama to time series data. { LagLlama}\xspace follows a z-score normalization at a window level, and tokenizes the input time series by extracting lagged features from the past values of the time series at different time lags. This allows the model to learn how past observations influence future ones. { LagLlama}\xspace is pretrained on 27 publicly available time series datasets from various domains. The model has about $\sim 2.5$M parameters.
\paragraph{ {Moirai}\xspace} woo2024unified Salesforce's {Moirai}\xspace is pretrained on $27$B observations from time series datasets across nine domains. {Moirai}\xspace uses multiple patch (window) size projection layers to capture temporal patterns across various frequencies. It comes in 3 different model sizes -- largest of which has about $311$M parameters, which is significant in size compared to earlier models.
\paragraph{ {TTM}\xspace} ekambaram2024ttms IBM's {TTM}\xspace (Tiny Time Mixer), as the name suggests, is a small {TSLM}\xspace with $\leq 1$M parameters that is pretrained on the Monash time series repository godahewa2021monash and LibCity libcitylong with $\sim 1$B data points. {TTM}\xspace is based on TSMixer architecture ekambaram2023tsmixer which uses MLP-based (multi-layer perception) mixing across time steps and features. This enables multivariate forecasting through efficient extraction of temporal and cross-variate patterns for time series forecasting. {TTM}\xspace first normalizes each sequence to have zero mean and unit standard deviation. Then it considers patches (windows) of varying length and varying resolution, since each dataset may have different optimal context to consider. {TTM}\xspace also applies downsampling on high resolution time series (e.g. seconds, minutes) to augment the training dataset.
\paragraph{ {Time-GPT}\xspace} garza2023timegpt1 {Time-GPT}\xspace was trained on a collection of publicly available time series, collectively encompassing over 100 billion data points. This training set incorporates time series from a broad array of domains, including finance, economics, demographics, healthcare, weather, IoT sensor data, energy, web traffic, sales, transport, and banking. Due to this diverse set of domains, the training dataset contains time series with a wide range of characteristics, ranging from multiple number of seasonalities, cycles of different lengths, and various types of trends.
\paragraph{ {TimesFM}\xspace} das2024decoder Google's {TimesFM}\xspace is a $200$M parameter {TSLM}\xspace. It is trained on largest corpus of 100 billion time points that includes both real and synthetic time series. {TimesFM}\xspace breaks time series in patches and learns to predict subsequent patches, and uses a standard normalization at patch level.
(ref) lists a few additional details on the various {TSLM}\xspaces described in this section. In particular, it provides information on when the model was released, i.e. when the pretraining sample ends, the list of all the data domains used in the pretraining phase of the model, and the size of training sample. It also flags whether each {TSLM}\xspace is univariate or multivariate.
Previous work focusing on macroeconomic forecasting has evidenced that models which perform particularly well are Bayesian VARs (BVARs) and Factor Models. See e.g. Stock:Watson:2006, Banburaetal2010, Carriero:etal:JBES:2016, CARRIERO2019JOE, KOROBILIS2019241, KOOP2019135, chan2022asymmetric. We therefore start from these models. We also include neural networks and regression trees in the comparison, as recent work by hyndman2018forecasting, HUBER202352, and Clark_etal_2023 show that in some instances allowing for non-linearities can help improve the accuracy of macroeconomic forecasts. Below we describe the specific choices we made in terms of the model specifications.
We consider the popular BVARs with natural conjugate Normal-inverted Wishart (N-IW) priors. This prior dates back to Zellner1971 and was later studied by KK1993,Kadiyala:Karlsson:1997. In a seminal paper, Banburaetal2010 show that it can be successfully applied to a very large cross section of macroeconomic data. Several contributions followed using this model and prior to handle large macroeconomic datasets.
Collect $N$ different variables in the vector $y_{t}=(y_{1t}\ y_{2t}\ ...\ y_{Nt})^{\prime }$, and write the Vector Autoregression or order $p$, $VAR(p)$, as:
where $t=1,\dots,T$ denotes time. Note that with $p$ lags and $N$ variables, each equation has $K=Np+1$ regressors. By grouping the coefficient matrices in the $N\times $ $K$ matrix $\Phi ^{^{\prime }}=[\Phi _{c}\ \Phi _{1}\ ...\ \Phi _{p}]$ and defining $x_{t}=(1\ y_{t-1}^{\prime }\ ...\ y_{t-p}^{\prime })^{\prime }\ $as a vector containing an intercept and $ p$ lags of $y_{t}$,\ the $VAR$ can be written as: $y_{t}=\Phi ^{\prime }x_{t}+\varepsilon _{t}. $ Next, after stacking the $t=1,\dots,T$ equations by column and transposing we can rewrite the $VAR(p)$ using more compact matrix notation:
where $Y=$ $[y_{1},..,y_{T}]^{\prime }$, $X=$ $[x_{1},..,x_{T}]^{\prime }$, and $E=[\varepsilon _{1},..,\varepsilon _{T}]^{\prime }$ are $ T\times N$, $\ T\times K\ $and $T\times N$ matrices.\
The conjugate N-IW prior elicits the prior on the coefficients conditionally on the error variance $\Sigma $:
The conditional posterior distribution of this model is also N-IW:
where $ \bar{\Phi}=\bar{\Omega}(\Omega _{0}^{-1}\Phi _{0}+X^{\prime }Y)$, $\bar{\Omega}=(\Omega _{0}^{-1}+X^{\prime }X)^{-1}$, $\bar{S}=S_{0}+Y^{\prime }Y+\Phi _{0}^{\prime }\Omega _{0}^{-1}\Phi _{0}-\bar{\Phi}^{\prime }\bar{\Omega}^{-1}\bar{\Phi}$, and $\ \bar{v}=v_{0}+T$. We elicit $\Phi _{0},\Omega_{0},S_{0},v_{0}$ as in the Minnesota tradition. To these priors, we add the "sum of coefficients" and "single unit root" priors of Doanetal1984 and Sims:1993:nber:chapter, which are both discussed also in Sims:Zha:IER:1998. These priors were motivated by the need to avoid having an unreasonably large share of the sample period variation in the data accounted for by deterministic components. All the priors described above ultimately depend on a small set of hyperparemeters, which we estimate by maximizing the marginal likelihood of the model, which is available in closed form. More details can be found in Appendix (ref).
This model can be used for very large cross sections because the Kronecker product in ((ref)) implies that the error variance $\Sigma$ enters the prior variance of the coefficients in a way that is symmetric and proportional to each equation. This fact, together with the fact that in a VAR every equation has the same regressors, implies that the posterior distribution of $p(\Phi, \Sigma)$ is a matricvariate-t. This presents several advantages as in this case i) a closed form solution exists for the marginal likelihood, ii) simulations can be performed via direct Monte Carlo (MC) sampling (as opposed to e.g. Markov Chain MC or Sequential MC), and iii) the operations required to perform such MC sampling are at most of order $O(N^{3})$. $\ $See Carriero_jae.2315 for a discussion. For these reasons, as shown by Banburaetal2010, the BVAR with conjugate prior can handle very large data sets with ease.\footnote{It is worth pointing out that one important limitation of the natural conjugate prior described in this section is that the Kronecker structure of the prior variance of the coefficients implies that the prior must be symmetric across equations. This in turn rules out cross-variable shrinkage, i.e. the possibility to shrink coefficients on lags of other variables more aggressively than those on own lags. chan2022asymmetric developed an alternative specification that allows for asymmetry in the prior, while maintaining conjugacy. We experimented with this alternative and found the results to be broadly similar; therefore, we did not include them in the paper. These results are available from the authors on request.}
Factor models are another class of models that has repeatedly shown to be particularly well suited for macroeocnomic forecasting. In this paper we use the implementation of McCracken:Ng:2016, which we now proceed to summarize. First, a set of static factors is estimated from the entire cross section of available data using Principal Components Analysis and the EM of Stock:Watson:JBES:2002 to balance the panel by filling in any missing values. Second, we use the extracted first factor to augment an autoregression of the $i$-th series with its lag values, i.e.:
We estimate this model via OLS, and rely on BIC to select the number of lags of $ \hat{f}_{1t}$ and $y_{it}$. The $h$-step ahead forecast is then computed as
Note that this approach (known as the "direct approach") is directly minimizing the $h$-step ahead forecast error, and it requires to perform a different regression for each forecast horizon.
VARs and factor models assume linearity between endogenous variables and their lags or lags of their linear combinations. Although these models have proven quite successful at forecasting macroeconomic time series, the linearity assumption could be overly restrictive and this could have a negative impact on forecast accuracy. To test this hypothesis, we also include in the comparison the Neural Network Autoregression (NNAR) model of hyndman2018forecasting and popularized through the forecast package published by the same authors. The NNAR model combines the structure of classical autoregressive models with the flexibility of feed-forward neural networks. In particular, the model is specified as: \[ y_{i,t} = f(y_{i,t-1}, y_{i,t-2}, \dots, y_{i,t-p}) + \varepsilon_{i,t}, \] where \( f(\cdot) \) is a nonlinear function represented by a single-hidden-layer feed-forward neural network with \( k \) hidden nodes, and \( \varepsilon_{i,t} \) is a white noise error term. The model is denoted as NNAR(p,k), analogous to an AR(p) model in traditional time series analysis. The NNAR model is based on a single hidden layer neural network trained using backpropagation, where we set the number of lagged inputs to \( p=12 \). Furthermore, to avoid overfitting, the model uses weight decay (L2 penalty) during training. Forecasts from the NNAR model are generated iteratively, and averaging over multiple networks trained with different initial weights, thereby capturing model uncertainty. Despite its simplicity compared to modern deep learning models, the NNAR model often performs competitively on short-term forecasting tasks, especially for time series with nonlinear dynamics and limited data.\footnote{We have also experimented with DeepAR salinas2020deepar, a sigificantly more sophisticated approach leveraging recurrent neural networks that trains the network using all available time series at once, but found it overall to not be more accurate than this simpler alternative.}
Lastly, we include in the pool of candidate models Bayesian Additive Regression Trees (BART). BART Chipman_etal_2010 models the conditional mean of the regression model by summing over a large number of trees which are, by themselves, constrained through a regularization prior. The main intuition behind the approach is that while each of these simple trees may only be able to explain a small fraction of the variation in the response variable, summing over a large number of them can describe the data extremely well. Pruser_2019, Huber_Rossini_2022, HUBER202352 and Clark_etal_2023 are recent examples of successful applications of BART with macroeconomic data.
For each series $i$, we implement BART by estimating the following model\footnote{We separate $y_{i,t-h}$ from the other predictors to avoid the model not recognizing that the lagged depedent variable is likely to have a non trivial role in predicting $y_{i,t}$. Our modeling choice is inspired by the approach of Lopez_et_al_WP_2025, who modified the standard BART to introduce a Minnesota-type shrinkage specification into the node splitting selection. We also consider the alternative model $y_{i,t}=f\left(X_{t-h}\right)+\varepsilon _{i,t}$ but we found it to perform significantly below our preferred specification.}
where $X_{-i,t-h}$ denotes all series at time $t-h$ other than $y_{i,t-h}$ and $g\left(X_{-i,t-h}\right)$ is a non-linear function obtained by summing over a larger number of regression trees,
where $g\left(X_{-i,t-h}\vert \mathcal{T}_j,m_j\right)$ identifies a single tree model with $\mathcal{T}_j$ denoting the tree structure and $m_j$ is the vector of terminal node parameters associated with $\mathcal{T}_j$. Following Chipman_etal_2010, we set $M = 250$ in our application and specify a prior on the tree structure along the lines they suggested. The $h$-step ahead forecast is then computed as
As with factor models, this approach requires a different regression for each forecast horizon.
We collect 120 monthly variables for the US spanning the period January 1959 to December 2022. The data, which are obtained from the Federal Reserve Economic Data (FRED) and are available at \href{https://fred.stlouisfed.org}{https://fred.stlouisfed.org}, cover a wide range of key macroeconomic variables that applied economists monitor regularly, such as different measures of output, prices, interest and exchange rates, and stock market performance. We provide a full list of the data and their transformations in (ref).
We use the first twenty-five year of data, January 1960--December 1984, to obtain initial parameter estimates for all the econometric models, which are then used to predict outcomes from January 1985 ($h=1$) to December 1985 ($h=12$). The next period, we include data for January 1985 in the estimation sample, and use the resulting estimates to predict the outcomes from February 1985 to January 1986. We proceed recursively in this fashion until December 2022, thus generating a time series of forecasts for each forecast horizon $h$, with $h=1,...,12$.
For the BVAR, point forecasts are obtained by taking the mean of the predictive densities. When $h>1$, the predictive densities are produced by iterating the model forward via simulation.\footnote{We set the number of lags in the BVARs to $p=6$.} For the factor models we first extract the first common factor, $\hat{f}_{1t}$. Next, for each variable and each forecast horizon, we run the factor-augment autoregression in (ref) and forecast up to $12$ months ahead. Similarly, for BART we re-estimate the trees for each variable set and sample, as the number of series in $X_{-i,t-h}$ changes with each variable set.
We proceed in a similar fashion with all {TSLM}\xspaces. More specifically, for each {TSLM}\xspace considered, starting with the first forecast date (December 1984), we feed the historical data to the pretrained model (when working with the univariate {TSLM}\xspaces we supplied one time series at a time, while with the multivariate {TSLM}\xspaces we supplied all time series within the specific model size at once). Next, we query the model to generate a zero-shot forecast for each of the variables provided, up to 12 steps ahead.\footnote{For all our experiments, we downloaded the open-sourced pretrained models and run them locally on our computer, using an Nvidia A6000 GPU with 48GB of memory. The only exception was {Time-GPT}\xspace, where we instead queried directly their API, with a separate query for each forecast date/time series pair.} We also include a benchmark approach which uses OLS forecasts from univariate AR(1)\ models.
We evaluate the predictive accuracy of the various models, for each of the variables considered and each forecast horizon. In particular, we measure the precision of the $h$-step-ahead point forecasts for model $i$ and variable $j$, relative to that from the univariate AR(1), by means of the ratio of Root MSFEs:
where $\underline{t}$ and $\overline{t}$ denote the start and end of the out-of-sample period, and where $e_{i,j,\tau +h}^{2}$ and $e_{bcmk,j,\tau +h}^{2}$ are the squared forecast errors of variable $j$ at time $\tau $ and forecast horizon $h$ associated with model $i$ ($i\in \left\{ \text{BVAR},\text{Factor model},\text{NNAR},\text{BART}\right\} \cup {TSLM}\xspace{}s$) and the AR(1) model, respectively. The point forecasts used to compute the forecast errors are obtained by averaging over the draws from the various models' $h$-step-ahead predictive densities. Values of $RMSFE_{ijh}$ below one suggest that model $i$ produces more accurate point forecasts than the AR(1) benchmark for variable $j$ and forecast horizon $h$.
In closing this Section it is worth mentioning that in this paper we are only focusing on point forecast accuracy. This choice is intentional. While there is a rich literature on density forecasts for macroeconomic variables, using {TSLM}\xspaces to construct density forecasts presents additional challenges, and we leave that to further research.
To begin, we focus on the overall forecasting performance of the various {TSLM}\xspaces under examination. Given that the evaluation sample we are considering include the Covid-19 period, to avoid conflating the overall picture with the large idiosyncrasies brought about by the pandemic, we focus first on the accuracy of the various models by stopping before the onset of the pandemic. That is, our evaluation sample stops in December 2019. Later on, we analyze how stable these results are over time, and also look at the accuracy of the various models over the most recent period, January 2020 to December 2022. (ref) provides a synoptic view of the forecasting performance across all 120 series we focused on, as measured by RMSFE ratios relative to the AR benchmark. Each box-plot shows the interquartile range of the distribution of relative RMSFEs for each of the {TSLM}\xspaces considered. The solid line within each box represents the median, while the whiskers represent the maximum and minimum relative RMSFEs that are not defined as outliers.\footnote{Outliers are defined as values that are more than 1.5 times the interquantile range away from the top or bottom of the box (i.e. from the lower and upper quartiles). } When the mass is concentrated above the line corresponding to $RMSFE=1$, this is an indication that the models do generally worse than the benchmark.
Outliers are not depicted for scaling reasons, and instead we separately report the maximum and minimum RMSFEs attained by each model in (ref) (together with their median and standard deviation). Indeed, it does happen on several occasions that for some series the {TSLM}\xspaces produce forecasts that are patently unreasonable. These forecasts would be automatically disregarded by the user and would call for a fine-tuning of the models. This result highlights that while zero-shot {TSLM}\xspaces may produce on average reasonable forecasts, they should not be used mechanically and do require careful monitoring.
Having said that, the pattern emerging from a careful inspection of (ref) is one of heterogeneity among the various {TSLM}\xspaces. Specifically, some {TSLM}\xspaces show a performance that is consistently worse than others. These are {TTM}\xspace-enhanced and {Time-GPT}\xspace, for which the box plots are invariably positioned higher than those of the remaining {TSLM}\xspaces, signaling higher RMSFEs. These models also perform poorly compared to the benchmark, for example {Time-GPT}\xspace has large part of the mass of the box-plot above 1 for all horizons, and {TTM}\xspace for the one-step-ahead horizon. { LagLlama}\xspace performs a bit better than these two, however it is also systematically outperformed by the remaining models.
The {TSLM}\xspaces showing the best forecasting performance are {Moirai}\xspace and {TimesFM}\xspace. As we discussed, {Moirai}\xspace comes in three sizes, and the evidence shows that the largest size performs the best, with a mass of RMSFEs being consistently below 1 at all horizons. Because of these considerations, in the remainder of this Section, we will only focus on these best performing {TSLM}\xspaces ({Moirai}\xspace-{large}\xspace and {TimesFM}\xspace), and compare their performance against the econometric models.
(ref) provides a summary of the forecasting performance of the two best {TSLM}\xspaces contrasted with the performance of the econometric models under consideration. As before, the box-plots represent the interquartile range of the distribution of RMSFEs (relative to the AR(1) benchmark) and the whiskers represent the maximum and minimum data points that are not flagged as outliers.
Focusing first on the econometric models, both the BVAR and the factor model are consistently showing a good performance relative to the AR model. This is a well known result, which has been obtained by several different studies using this dataset.
As already shown in (ref), {Moirai}\xspace-{large}\xspace and {TimesFM}\xspace also perform consistently better than the AR benchmark, and (ref) now shows that the RMSFE distribution is similar to those obtained with the econometric models. Interestingly, while all models at almost all horizons have a median that is below the threshold of one, there is a tendency for the distribution of the RMSFEs from {TSLM}\xspaces to have a more pronounced right tail, signaling an increased likelihood of observing large forecast errors. Instead, econometric methods produce RMSFE distributions that are more left-skewed, signaling a consistent better performance compared to the AR(1).
Turning to the two non-linear econometric models, the evidence is mixed. The performance of BART is consistently worse than that of the benchmark. The simple NNAR model produces better forecast than BART, and is overall slightly better than the AR(1) benchmark. At 1-step ahead, the performance of NNAR is similar to that of the TSLMs. However, at longer horizons this model is outperformed by both TSLMs and the linear econometric models. This suggests that while non-linearities do play an important role, TSLMs seem to have an edge at capturing them, over the simpler non-linear models we considered.
As before, additional descriptive statistics including the maximum and minimum relative RMSFEs (including outliers) are reported in (ref). The linear econometric models (BVARs and factor models) do not present any extreme cases in which forecast errors are a multiple of those obtained by the AR benchmark, which it is instead the case for the {TSLM}\xspaces results shown in (ref), and - to a lesser extent - for the nonlinear models in the last two rows of (ref). Moreover, it is evident that the RMFSEs coming out of the econometric models are less dispersed. To sum up, the picture is one in which BVARs and factor models seem to offer a more stable and reliable performance, when compared to zero-shot {TSLM}\xspaces and other simpler nonlinear models.
Note that so far we have not considered the statistical significance of the various forecasts. (ref) presents results on the statistical significance of the various forecasts (relative to the AR(1) benchmark), focusing for ease of exposition on a subset of key macroenomic variables. These results presented in there show that for several series the improvements of both the econometric models and the TSLMs over the AR benchmark are statistically significant.
The results presented so far gave an overall picture of the accuracy of the various models across the different forecast horizons, focusing on the evaluation sample January 1985 to December 2019. In this section, we focus on the post Covid-19 period sample, Jan 2020 to December 2022. Also, we exclude from our analysis the forecasts made between March and June 2020, to reduce the impact on the results of the extreme outliers right around the onset of the pandemic.\footnote{The choice of excluding from the sample the most problematic Covid-19 related observation is based exclusively on simplicity. Still, there is evidence that simple approaches such as this might work well, see SchorfheideSong:2021:nber:var. One could also consider more sophisticated ways to deal with the Covid-19 observations such as CCMM:RESTAT:svo or LenzaPrimiceri:2022:jae:VARafter2020.}
(ref) compares the forecast accuracy of the various {TSLM}\xspaces, and (ref) considers the best performing {TSLM}\xspaces against the various econometric models.
Compared to what we shown in (ref) and (ref) there is an overall deterioration in forecast accuracy for all the models, which all struggle in overperforming the benchmark. Within the LLMs, only {Moirai}\xspace-{large}\xspace stands out, with a performance comparable with that of the factor model and the BVAR. In this sample the econometric models tend to make at times large forecast errors, mainly during the second half of 2020. For the factor model it happens especially for interest rates and spreads.
The usual caveat applies that, as shown in (ref), namely that the {TSLM}\xspaces include the pandemic period in their training sample (and a few also include many of the variables we set out to forecasts), and this makes it harder to pin down to which extent this forecasting performance could have been achieved in real time.
To better understand the results presented so far, this Section explores three key aspects of the {TSLM}\xspaces and how these impact the models' forecast performance. In Section (ref) we consider the effects of possible "data leakage", which as we mentioned before can spuriously tilt the forecast accuracy scale in favor of the TSLMs. In Section (ref) we show that highly persistent series are particularly problematic for TSLMs to forecast. Finally, in Section (ref) we consider the impact of fine tuning on the {TSLM}\xspace forecasts, i.e. updating the model parameter estimates to reflect more closely the information and data patterns of the macroeconomic time series contained in the FRED-MD database.
In the Introduction, we emphasized that it is not straightforward to assess the real-time forecast accuracy of {TSLM}\xspaces, mainly because these models are trained on data lacking timestamps. Thus, previous results may overstate {TSLM}\xspace forecasting ability, introducing a potential look-ahead bias due to some of the training data postdating the forecast date. To further complicate things, we saw in (ref) that some of the {TSLM}\xspace models do include in the training set the FRED-MD database, which means they potentially have access to the perfect forecast. Although retraining the {TSLM}\xspace models in real time is difficult and costly, we can perform the opposite experiment of giving the same unfair advantage to the econometric models. This is the experiment we describe in this subsection.
Specifically, we re-estimated the econometric models using the entire sample available, up to December 2022. This date precedes or coincides with the dates in which the TSLMs were trained (based on the year of publication of the corresponding papers, during 2024). While this comparison is not perfect, it can provide a first order approximation of the potential impact of using different information sets in the model training phase. (ref) shows the results of this experiment, where the BVAR and Factor model RMSFEs are replaced by their full-sample counterparts. {Moirai}\xspace and {TimesFM}\xspace results, taken from (ref), are included to make the comparison easier.
The results are revealing. The forecasting performance of BVAR and Factor models is dramatically enhanced by the use of the extra information, with their respective box plots lying entirely below one. In other words, these models consistently beat the AR benchmark across most series and horizons. They also perform significantly better than the {TSLM}\xspaces, which appeared to perform comparably to the econometric models only when the latter were estimated *without* the benefit of hindsight.
As described above, many of the macroeconomic series are transformed before entering the estimation stage of the models. Looking at the results shown so far, there appears to be some indication that {TSLM}\xspaces perform relatively better when the degree of persistence of the series they utilize is relatively low. To further investigate this aspect, we break down the previous results by separately looking at series with different degrees of persistence. Specifically, (ref) shows results for variables for which the one lag partial autocorrelation is below 0.7, while (ref) shows results for variables for which the one lag partial autocorrelation is above 0.7. The latter variables are of course highly persistent.
For the econometric models, the pattern emerging from this particular cut of the results is in line with expectations. Specifically, VARs perform relatively better when estimated with more persistent variables (likely because keeping the variables in levels allows for common trends and co-integration) while factor models perform relatively better when estimated with less persistent variables (likely because the estimation of the factors relies on stationarity). This is evident in the graphs, where the factor model box-plots are consistently below 1 in (ref) while the BVAR box-plots are consistently below 1 in (ref).
Turning the attention to {TSLM}\xspaces, for series with low to moderate persistence, they seem to perform relatively better, still on par with BVARs and factor models. The overall picture changes starkly when considering highly persistent variables. In these cases, one can observe a definite increase in the variation of the performance of {TSLM}\xspace. Interestingly, the same pattern applies to the non-linear econometric models, NNAR and BART.
To give an example of the extent of variation in performance, (ref) shows the time series of out-of-sample forecasts produced by the factor model, the BVAR and {TimesFM}\xspace for a few selected variables. While for the 1-step ahead horizon the forecasts are quite similar (left side panels), large differences emerge at the 12-step ahead horizon (right side panels).
For Housing Starts (HOUST), the 12-step ahead forecasts produced by {TimesFM}\xspace are remarkably good, as this model is able to almost perfectly predict the pronounced slowdown that happened during the 2007 financial crisis and the ensuing recession. This results in an astonishing 50 percent improvement in RMSFE compared to the AR benchmark. Instead, the econometric models only pick up this slowdown with a delay, and the factor model even fails to recognize the depth of the trough. Of course, one must keep in mind that {TimesFM}\xspace includes this recessionary time period in its training sample, so it seems to be on some level aware that a change in time series behavior is to be expected within these dates. For weekly hours (CES0600000007) {TimesFM}\xspace outperforms the factor model but underperforms the BVAR. For variables such as the spread between the 10 year interest rate and the Federal Funds rate (T10YFFM) and the spread between corporate bonds and the Federal Funds rate (BAAFFM), the performance of {TSLM}\xspaces is subpar, featuring entire periods of "hallucination" in the second half of the 80s and in 2017-2019, resulting in RMSFE losses on the order of 15-20 percent.
Up until now, all {TSLM}\xspaces results have been based on zero-shot predictions of the original pretrained models. As discussed in (ref), alternatively one could consider the option of fine-tuning the {TSLM}\xspaces, i.e. updating the pretrained model's parameters on a task-specific dataset, with the goal of improving forecasting performance. Broadly speaking, to implement fine-tuning one could follow one of two approaches, (i) Continued-pretraining or (ii) Parameter-efficient fine-tuning.\footnote{See han2024parameter for a detailed review.} Continued-pretraining involves further training the model on a domain-specific dataset starting with pretrained parameters, and allowing all model parameters to adapt to the characteristics of the given data. This can lead to overfitting when the fine-tuning data is small. Parameter-efficient methods instead freeze some of the original model's parameters and introduce trainable low-rank matrices (e.g. low-rank adaptation (LoRA) methods as in hu2022lora), significantly reducing the number of trainable parameters, addressing overfitting, and requiring comparatively less computational resources.
Here, we selected the two best models from our zero-shot experiments -- {TimesFM}\xspace and {Moirai}\xspace-{large}\xspace -- and fine-tuned them on {FRED-MD}\xspace data. Note that some of the pretrained {TSLM}\xspace{s}, including {Moirai}\xspace, are trained on datasets that overlap with the {FRED-MD}\xspace. However, the influence of any single dataset in the pretraining phase gets diluted within the vast and diverse pretraining corpus. Consequently, the models learn general time series patterns, rather than specializing in the specific characteristics of {FRED-MD}\xspace. To make a {TSLM}\xspace more dataset-specific, fine-tuning is required. The main challenge we face in this context is that FRED-MD is a relatively small dataset, which may lead to overfitting problems.\footnote{kumar2022fine discusses how in these situations the model may also lose the knowledge acquired during pretraining, leading to the so-called catastrophic forgetting.} To address these challenges, we employ a simple parameter-efficient fine-tuning strategy as opposed to continued-pretraining. In particular, we freeze the transformer layers of the two {TSLM}\xspaces, thereby reducing the number of trainable parameters.\footnote{kumar2022fine refers to this as Linear Probing.} Note that freezing the pretrained model's weights (parameters) is a special case of LoRA, which additionally injects trainable rank decomposition matrices into each layer of {TSLM}\xspace. After freezing the transformer layers, the number of trainable parameters reduced to approximately $\leq 10$M for both {TimesFM}\xspace (instead of $200$M) and {Moirai}\xspace-{large}\xspace (instead of $311$M), primarily part of the final layers.\footnote{In the case of {Moirai}\xspace, we also consider fine-tuning using the continued-pretraining strategy, as this was the recommended approach provided by the authors in the model's repository, but in this case the model ended up severely overfitting and producing poor out-of-sample forecasts.} Throughout, to keep computational costs manageable, we chose to fine-tune each model once every 12 months, and the resulting models were then used for zero-shot forecasting over the subsequent 12 months.\footnote{Within each fine-tuning round, the training {FRED-MD}\xspace data was split into an 80:20 ratio, with 20% reserved for validation. We trained these models for 200 epochs, utilizing early stopping with a patience of 5 to mitigate overfitting. To further counter overfitting and ensure stable training on our limited dataset, we reduced the learning rate, preventing large weight updates.} We run the fine-tuning experiment on a stock server equipped with a single NVIDIA A6000 GPU. In terms of wall-clock time, fine-tuning took about 10 hours for each of the two models. Hence, comparatively, fine-tuning remains significantly more resource-intensive than the BVARs and factor models we considered, which can be trained on standard CPUs in a fraction of the time.\footnote{We also considered the continued-pretraining option for the same models, and in that case the total computing time was approximately 90 hours.}
(ref) shows the comparison of zero-shot and fine-tuned performance for {TimesFM}\xspace and {Moirai}\xspace-{large}\xspace. We observe that fine-tuning yields results that are similar or slightly better than the zero-shot performance for both models. This result overall isn't particularly surprising for our setup: a large model and a small dataset. While fine-tuning offers the potential for adapting models to a domain, it comes with its own set of challenges, such as hyperparameter optimization and increased computational resource (e.g. GPUs) requirements, thus needs careful consideration. Given the marginal performance gains achieved through fine-tuning in our context, the inherent "plug-and-play" simplicity of zero-shot forecasting presents itself as a potentially more practical approach.
The results so far show that although the econometric model consistently outperforms the AR benchmark, in some cases {TSLM}\xspaces can be very accurate. At the same time, in a number of cases the performance of {TSLM}\xspaces can be significantly sub-par. One way to rationalize these results is that while TSLMs are capable of exploiting vast datasets, they still lack the distilled domain knowledge built from decades of macroeconomic analysis. Still, in some cases, their ability to capture nonlinearities in the data can be quite advantageous.
In this section, we consider an approach that can help leverage the best of both worlds: that is, exploiting the domain knowledge and simplicity of the econometric models while relying on {TSLM}\xspaces to capture nonlinear behavior missed by the traditional approaches.
Our proposal is to build a simple hybrid approach. First, an econometric model (we use the BVAR) is fit to the data and the in-sample residuals are computed. Second, the residuals from the first step are modeled using a {TSLM}\xspace. Finally, forecasts are computed from both models and combined. Hence, the final forecast adjusts the econometric prediction by adding the predicted residual component, effectively correcting systematic biases and capturing nonlinearities.
Formally, let $y_{i,t}$ denote the target variable of interest. The procedure comprises the following steps:
This approach retains the interpretability of the BVAR while flexibly capturing nonlinear residual dynamics through the foundational architecture.
(ref) and (ref) present the results of this analysis. In particular, (ref) shows results for the original BVAR plus the two best-performing {TSLM}\xspaces, {Moirai}\xspace and {TimesFM}\xspace, and finally two hybrid versions combining the BVAR with both {TSLM}\xspaces. The hybrid approach shifts the TSLM error distribution downward, placing most of its mass below one—the threshold for outperforming the AR benchmark.
Moreover, (ref) shows that the maximum relative RMSFEs from the TSLMs are greatly reduced by the hybrid approach: at one step ahead, the error of {Moirai}\xspace decreases from 1.436 to 1.139, and that of {TimesFM}\xspace from 1.482 to 1.150. These figures are very close to the maximum error of the BVAR, which is 1.158. Thus, the use of econometric models in the first step substantially reduces the risk of hallucinations and other aberrant forecast behaviors. Incorporating the domain knowledge of econometric models helps TSLMs across the board by lowering forecast errors for all series.
When comparing the hybrid versions with the BVAR (first box on the left in the figure), one can notice that adding to this model the predictions of the residuals via TSLMs does not adversely affect the right tail of the distribution, as there are no forecasts that worsen as a result. Instead, the addition has a positive effect on the left tail: for those forecasts that were already improving over the AR benchmark, the magnitude of the improvement increases. In (ref), the best relative RMSFE from the BVAR model shows improvements ranging from 5% to 12% depending on the forecast horizon. For example, at 1-step ahead the best relative RMSFE drops from 0.827 to 0.734 or 0.720 when the BVAR is used in conjunction with {Moirai}\xspace and {TimesFM}\xspace, respectively.
Overall, the hybrid approach improves upon both the econometric model and the TSLMs used in isolation. Using the econometric model in the first step prevents TSLMs from producing subpar forecasts, while using the TSLMs on the residuals enhances the performance of the econometric model. This suggests that the domain knowledge of econometric models is robust and reliable, but can be further improved using TSLMs to capture nonlinearities.\footnote{While these results are encouraging, one must not forget the caveat that the performance improvement form TSLMs we observed here might be due to the fact it has a spuriously superior information set, as discussed in Section (ref).}
In this paper, we have provided a comprehensive evaluation of the forecasting performance of Time Series Language Models ({TSLM}\xspaces) in predicting macroeconomic variables. While recent advances in machine learning have led to the development of these models as promising alternatives to traditional econometric methods, our results indicate that their performance remains mixed. Specifically, only two of the five models we evaluated, Salesforce's {Moirai}\xspace and Google's {TimesFM}\xspace, consistently outperform a simple autoregressive benchmark. Furthermore, these same models struggle to deliver consistently superior forecasts compared to established macroeconomic forecasting methods such as Bayesian Vector Autoregressions (BVARs) and Factor Models. Our findings therefore suggest that while {TSLM}\xspaces can offer valuable insights, they are not yet a clear replacement for state-of-the-art econometric models.
A key challenge in applying {TSLM}\xspaces to macroeconomic forecasting is the lack of control over their training data. For example, many of these models are pretrained on the {FRED-MD}\xspace dataset, hence already contain the macroeconomic series that are the focus of our forecasting exercise, introducing potential biases and making it difficult to conduct a clean pseudo out-of-sample forecasting exercise. Furthermore, practical challenges and computational constraints make retraining these models in real time a non-trivial task. Taken together, these issues raise important concerns about the role of {TSLM}\xspaces in economic forecasting, particularly in comparison to existing time series methods such as BVARs and factor models which have proven extremely flexible and successful in these same settings.
Despite these limitations, our results indicate that {TSLM}\xspaces exhibit some promising features, particularly in capturing nonlinearities and adapting to evolving economic conditions. Their relatively stronger performance in the post-Covid-19 period suggests that they may have advantages in handling structural breaks and rapidly changing economic environments.
With this in mind, we proposed a TSLM-augmented BVAR (TSLM-BVAR) hybrid approach leverages the complementary strengths of econometric models and foundational time series learning. By first generating forecasts with a BVAR and then modeling the residuals using TSLMs, the method captures nonlinear patterns missed by traditional models while retaining the interpretability and domain knowledge of econometrics. Empirical results show that this hybrid procedure consistently reduces forecast errors compared to either component in isolation, particularly mitigating extreme errors in TSLM predictions. Overall, the TSLM-BVAR approach demonstrates that integrating structured econometric information with flexible machine learning models can substantially improve forecasting performance.