Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
77,893 characters · 12 sections · 51 citation commands
Fake Date Tests: Can We Trust In-sample Accuracy of LLMs in Macroeconomic Forecasting?
\thispagestyle{empty} \doublespacing
Keywords: large language models, macroeconomic forecasting, lookahead bias, context bias \\
JEL codes: C12, C52, C53 \\
Large language models\footnote{In this paper, we will not distinguish between LLMs, LRMs (large reasoning models), and multimodal models that can generate text, and will refer to all such models as LLMs, unless such a distinction is explicitly required somewhere.} (LLMs) and LLM agents are becoming an essential tool in many areas of human activity. Among other things, they can edit and write texts, generate code in various programming languages\footnote{See, e.g., the \href{https://openai.com/index/introducing-gpt-5/}{GPT-5} model introduced by OpenAI.}, quickly search for and synthesize information from a large number of sources\footnote{See, e.g., the \href{https://openai.com/index/introducing-deep-research/}{Deep Research} mode in ChatGPT.}, and act as tutors\footnote{See, e.g., the \href{https://openai.com/index/chatgpt-study-mode/}{Study mode} in ChatGPT.}. These models are developing at a rapid pace; in the top-10 models on Chatbot Arena (Chiang et al. Chiang2024), which is one of the most popular resources for comparing LLMs, it is rare to find models that were released more than two months ago.
Macroeconomists are also increasingly paying attention to these models. They are a tool for automating routine researcher tasks (see Korinek (Korinek2023, Korinek2024, Korinek2025)), an object of influence on economic growth and policy (see Acemoglu Acemoglu2025; Agrawal et al. Agrawal2025), and a tool for economic modeling and forecasting (see Bybee Bybee2025; Li et al. Li2024; Lin et al. Lin2025; Lopez-Lira and Tang Lopez-Lira2025; Kazinnik and Sinclair Kazinnik2025; Hansen et al. Hansen2025; Zarifhonarvar Zarifhonarvar2026). It is forecasting, or rather the feasibility of testing the quality of LLMs' forecasts, to which this paper is dedicated.
The forecasting properties of macroeconomic models, whether statistical ones (such as Bayesian vector autoregressions and their variations (see Giannone et al. GLP2015)) or structural ones (such as dynamic stochastic general equilibrium models (see Diebold et al. DSS2017)), are usually tested in pseudo-real time. A model is trained on data available up to time period $t$, then a forecast is made for one or several future periods, after which the training boundary is shifted to period $t+1$, and the process repeats. This sequence of steps is designed to artificially simulate how the forecasting process would be organized in the past, among other things ensuring the model's lack of access to subsequent data unavailable at the time of forecasting.
Modern LLMs are typically not trained dynamically; rather, they fix a (sufficiently large) dataset at the pre-training stage, followed by the use of various techniques (such as supervised finetuning and reinforcement learning) for finer model tuning, after which the model weights remain fixed (see, for example, DeepSeek-AI DeepSeekR12025). Using such trained LLMs for retrospective accuracy evaluation, similarly to how it is done with classical macroeconomic models, is associated with several challenges.
The first challenge is related to excluding the model's ability to “look” into the future. The simplest and most obvious way to solve this problem is to use models only to produce forecasts for periods that could not possibly be included in the training datasets, i.e., for periods after the model's release date\footnote{We deliberately do not use the term “model's cutoff” herein, as it is quite vague and often fails to reflect the date beyond which the model has no knowledge (see the example with a cutoff from Paleka et al. Paleka2025).}. Despite the fact that this method is correct, there is currently not much macroeconomic data available to researchers that was released after the publication of LLMs themselves to statistically evaluate the forecast quality with confidence. Instead, to enable retrospective quality evaluation, researchers often resort to a trick where within a prompt\footnote{Text query.} they ask the model not to use any data after the date at which the forecast is being made (see, for example, Faria-e-Castro and Leibovici Faria-e-Castro2024; Hansen et al. Hansen2025; Tomáš et al. Tomas2025). Although such filters work well in some tasks (see Thaker et al. Thaker2024), a number of studies show that their application may not fully eliminate data leakage from the future (see Liu et al. Liu2025), which can in turn create lookahead bias in retrospective accuracy evaluation (see Sarkar and Vafa Sarkar2024; Ludwig et al. Ludwig2025). A solution to this problem could be dynamic model training, similar to what He et al. He2025 do in their study, building a sequence of language models whose knowledge is limited to a specific year. However, at the current stage of technology and algorithm development, this procedure is quite costly, and as far as we know, all such studies are limited to models that are several orders of magnitude smaller than modern LLMs, which naturally affects their quality.
The second challenge relates to the amount of information available at the time of forecasting. The main goal of evaluating retrospective accuracy is to obtain an estimate of how the model will behave in the future (see Ng Ng2018). This can be an estimate of absolute accuracy or accuracy relative to other models. In the case of LLMs, even in the absence of lookahead bias and with standard assumptions about the representativeness of historical data, the evaluation of retrospective accuracy may not reflect the model's quality in the future. This can be due to differences in knowledge about what is happening in the world at the time of forecasting. Thus, in the time periods included in the training sample, the model often knows additional context beyond the information from the prompt given by the researcher, such as, for example, detailed descriptions of economic policy measures, medical reports, rhetoric from different economic actors, and so on. As time moves away from the model's release date, this context begins to disappear, which can create an additional bias. Adding relevant up-to-date information, for example, through the ability to use internet search tools, as done in the \href{http://aiinflationexpectations.org/}{AI Inflation Expectations} project, or other sources may partially mitigate the emerging bias; however, the context may still be incomplete. Moreover, inaccurate implementation can cause additional biases similar to those described in the example using the Google search engine in Paleka et al. Paleka2025.
In this paper, we propose a family of prompt sensitivity tests based on the idea that in the case of expected LLM behavior, the results of macroeconomic forecasts should be insensitive to certain variations in the prompt. In particular, two tests, which we call fake date tests, and which we focus on in detail in this paper, can in some cases detect biases arising in the context of the two aforementioned challenges. It should be noted that passing the proposed tests does not guarantee the absence of these biases; however, failing them should cast serious doubt on the validity of the methodology used for retrospective accuracy evaluation.
Fake date test I examines the presence of lookahead bias by constructing two forecasts based on the same information provided to the LLM within the prompt, except for the forecast date. The first one is constructed for a forecast date that corresponds to the statistics provided and an information cutoff date that is sufficiently far in the past from the forecast date to avoid bias associated with the LLM having an up-to-date context. The second forecast differs from the first one in that the forecast date (but not the information cutoff date) is shifted into the future and is built far beyond the model's release date. A mismatch in the distributions of these forecasts signals that the LLM is unable to follow the instruction not to use any information after the cutoff date.
In Fake date test II, the cutoff date is moved so that it coincides with the period of the first forecast. Assuming that Fake date test I is passed and the information is cut off correctly, a mismatch in the forecasts for Fake date test II signals the presence of bias associated with the LLM having an up-to-date context. However, if Fake date test I is not passed, such a mismatch in the forecasts indicates the presence of at least one of the two aforementioned types of bias.
The results obtained on several modern LLMs demonstrate that none of the tested models with the fairly strict prompt used in this study and the statistics provided has passed Fake date test I or Fake date test II. This finding raises doubts about the validity of using the in-sample accuracy to evaluate the forecast quality of these models (particularly when using the specified prompt). Thus, despite their simplicity, our tests are quite powerful tools for detecting biases that may arise in LLM-based macroeconomic forecasting.
Related literature. Our contribution to the literature can be divided into two strands. The first strand is the growing literature on macroeconomic forecasting and modeling economic expectations based on LLMs. Examples of forecasting papers include Faria-e-Castro and Leibovici Faria-e-Castro2024, Tomáš et al. Tomas2025, André et al. Andre2025 and Hansen et al. Hansen2025. All these papers use in-sample forecasts to estimate the accuracy of the models, and for this type of research, our tests provide a simple and easy-to-use tool for assessing the validity of the quality evaluation procedure. Related papers on modeling economic expectations based on a small amount of statistical information, such as Lin et al. Lin2025 and Zarifhonarvar Zarifhonarvar2026, although they do not make direct predictions, may nevertheless contain similar biases, and our tests may be useful for them to investigate in-sample biases and context bias near the model's release date. This could involve direct testing for each individual in the sample of economic agents modeled using LLMs, where the tests can be applied directly but would require significant token costs, or a modification of the test using aggregated expectations, which can be obtained through a simple extension of the test to introduce heterogeneity\footnote{This is also relevant for Hansen et al. Hansen2025 in the case of heterogeneous characteristics of forecasters.}.
Second, we make a methodological contribution to the study of biases arising from in-sample forecasting using LLMs, an emerging topic of interest to economists. In their seminal paper, Sarkar and Vafa Sarkar2024 design an event-based test for lookahead bias to determine whether LLMs can accurately predict the outcomes of close elections and determine risk factors from earning calls. In a recent paper, Crane et al. Crane2025 study LLMs' recall of U.S. macroeconomic data, and find that models return a mix of vintages for real GDP growth. Based on these findings, they formulate a regression-based test for the presence of lookahead bias. This test is based on a strong assumption that the expectations provided by the Survey of Professional Forecasters are rational. In contrast to these papers, our Fake date test I is more universal because it can signal the presence of lookahead bias in any time period and for any time series without such strong assumptions as the rationality of expectations. Finally, there are papers by Didisheim et al. Didisheim2025 and Lopez-Lira et al. Lopez-LiraTangZhu2025 who study the broader problem of memorization. They propose prompt-based tests to evaluate the ability to recall the values of economic and financial variables on specific dates. Our tests, by design, focus on real-time forecasting applications where various types of bias remain a significant concern.
The rest of the paper is organized as follows. Section (ref) describes the testing methodology. Section (ref) discusses the interpretation of the proposed tests and their limitations. Section (ref) presents the results of testing several modern LLMs. Section (ref) concludes.
This section describes the methodology proposed for testing biases that might arise in macroeconomic forecasting. First, all necessary notations are introduced, and then a family of prompt sensitivity tests and two versions of the fake date test for detecting the biases described in the Introduction are proposed.
It is assumed that the forecast is built using an LLM trained on information set $I_T$, which is denoted as $\mathcal{M}_T$ in this paper. The model takes a user-provided prompt, $x$, as an input and returns an output, $y = \mathcal{M}_T(x,\varepsilon)$, where $\varepsilon$ is a stochastic component that introduces randomness. This randomness can be attributed to the model's sampling parameters (see Bishop and Bishop Bishop2024, p. 387) and to the implementation features\footnote{Despite the fact that in theory LLMs should return a deterministic response at zero temperature, modern implementation methods often do not provide such a guarantee (see He and Thinking Machines Lab He2025nondeterminism).}.
The prompt itself in the task of macroeconomic forecasting can be formulated in different ways, but usually includes several practically mandatory components:
Thus, the input prompt for the forecast has the form:
where $f$ is a function that combines all prompt components together. An example of one of the prompts that will be used later in Section (ref) is presented in Figure (ref).
The LLM's response ($y_{forecast}$) usually consists of an explanation ($y_{explain}$)\footnote{This includes reasoning in LRMs.} and a numerical value or a set of numerical values ($y_{answer}$), as shown in Figure (ref). For the purposes of the subsequent test, the most interesting component of the response is $y_{answer}$; therefore, without losing significant information, we will assume that $y_{forecast}$ consists only of $y_{answer}$.
The idea of the family of prompt sensitivity tests is based on the fact that the expected behavior of an LLM in macroeconomic forecasting should be insensitive to certain changes in the prompt.
Formally, this can be written as follows. Suppose there are $K$ prompts with parameters:
For each of these prompts, $N$ forecasts are sampled:
where each forecast
The null hypothesis involves testing a set of assumptions $\mathcal{H}$, in the presence of a number of other assumptions $\mathcal{A}$, which are considered true. If the set of assumptions $\mathcal{H}$ is satisfied, then for functions $s$ and $h$, which are responsible for aggregating the characteristics of forecast distributions and comparing them respectively:
To detect lookahead bias and context bias, we propose to sequentially conduct two tests (which we further refer to as Fake date test I and Fake date test II), comparing the forecasts with real and fake forecast dates.
Fake date test I evaluates the model's ability to follow the instruction not to use data after the cutoff boundary and is considered a proxy test for detecting lookahead bias. To check this, for each date $t_{retro}$ that is planned to be used when calculating retrospective accuracy, we calculate $N$ forecasts $y^1_{forecast}$, which only differ in the values of the realization of random variables $\varepsilon^1_n$, and approximate the forecast distribution:
where the date for which the forecast is made ($t_{forecast}$) is $h$ time periods ahead of the forecast date ($t_{current}$)\footnote{In the case of forecasting for several horizons within one prompt, $h$ is a set of horizons.}, and the information cutoff date ($t_{cutoff}$) is $d$ periods in the past from the forecast date ($t_{current}$). The forecast is built using information about the economic situation ($s_{retro}$), which corresponds to time $t_{retro}$, fixed textual description ($l$), and variables for the forecast ($v$).
The only parameter that differs from typical prompts used in macroeconomic research on forecasting with LLMs is the information cutoff date, which is shifted to the past. This is done to maximally exclude the influence of additional information beyond $s_{retro}$ (additional economic context) affecting the forecast and to focus solely on lookahead bias.
In addition to forecast ((ref)), a forecast $y^2_{forecast}$ is generated for a fake date $t_{fake}$:
In fact, this forecast differs from the previous one only in changing the forecast date ($t_{current}$) and the date for which the forecast is made ($t_{forecast}$). These dates are shifted in parallel beyond the model's release date. Examples of two prompts used later in Section (ref) are shown in Figure (ref).
To formalize the test, the following two assumptions are made:
(A1) Assumption 1. The distribution of the forecast with cutoff date $t_{cutoff}$ is measurable\footnote{The measurability of the forecast here and further is understood in the sense that for a fixed $\varepsilon$, the value $\mathcal{M_T}(f(\cdot), \varepsilon)$ is measurable.} with respect to the $\sigma$-algebra generated by the union of $s_{current}$ and $I_{t_{cutoff}}$ (the intersection of $I_T$ and all information available up to time $t_{cutoff}$).
(A2) Assumption 2. For two forecasts with cutoff boundary $t_{cutoff}$, measurable with respect to the $\sigma$-algebra generated by the union of $s_{retro}$ and $I_{t_{cutoff}}$, and differing only in the forecast date ($t_{current,1} \geq t_{cutoff}+d$ and $t_{current,2} \geq t_{cutoff}+d$, where $t_{current,1} \neq t_{current,2}$) and the date for which the forecast is made ($t_{forecast,1}=t_{current,1}+h$ and $t_{forecast,2}=t_{current,2}+h$, where $t_{forecast,1} \neq t_{forecast,2}$), the distributions coincide.
Assumption 1 formalizes the absence of lookahead bias and requires that only information up to the cutoff boundary and the information about the state of the economy specified in the prompt be used in the forecast. Assumption 2 requires that any two forecasts, sufficiently distant in time from the information set available to the LLM (except for the information about the state of the economy specified in the prompt), do not differ. This assumption can be interpreted as follows: the LLM should be insensitive to a specific date outside its information set (see more discussion in Sections (ref) and (ref)), and this information set should only be used to build general economic principles relevant to the forecasting task, not to form context regarding the forecast date. However, strictly speaking, this is merely what we tried to embed in Assumption 2 and these conditions may not necessarily be met, compensating each other to satisfy the assumption (see more discussion in Section (ref)).
Given these two assumptions, we can formulate Fake date test I.
Proposition 1 (Fake date test I). When $\mathcal{A} = \{A2\}$ and $\mathcal{H} = \{A1\}$ are satisfied, the distributions of forecasts $y^1_{forecast}$ and $y^2_{forecast}$ must coincide.
Proof. See Appendix (ref).
Proposition 1 means that if the distributions of $y^1_{forecast}$ and $y^2_{forecast}$ do not coincide, the model fails to correctly follow the instruction to cut off information. Empirically, for each date this can be tested using any test for comparing distributions, for example, a permutation test with the Kolmogorov-Smirnov statistic (choosing the cumulative distribution as $s$, and the test statistic as $h$), or tests for comparing the sample characteristics of distributions, for example, a test for equality of means ($s$ is the mean of the distribution, and $h$ is the test statistic). Multiple comparisons for all periods can also be made using classical techniques, for example, Bonferroni correction.
Both assumptions can be selected as $\mathcal{H}$ in the test (for example, $\mathcal{A}=\{\}$ and $\mathcal{H}=\{A1,A2\}$), which will weaken its interpretation in terms of lookahead bias; however, as is further discussed in Sections (ref) and (ref), a number of supporting experiments show that the influence of not satisfying Assumption 2 for the tested LLMs is potentially not as great as the influence of Assumption 1.
Fake date test II is similar to the first one and differs only in that depth $d$ in formulas ((ref)) and ((ref)) is taken to be zero (see example in Figure (ref)). Assumption 1 is preserved, while the depth in Assumption 2 is taken to be zero, i.e., it becomes:
(A2$'$) Assumption 2$'$. For two forecasts with cutoff boundary $t_{cutoff}$, measurable with respect to the $\sigma$-algebra generated by the union of $s_{retro}$ and $I_{t_{cutoff}}$, and differing only in the forecast date ($t_{current,1} \geq t_{cutoff}$ and $t_{current,2} \geq t_{cutoff}$, where $t_{current,1} \neq t_{current,2}$) and the date for which the forecast is made ($t_{forecast,1}=t_{current,1}+h$ and $t_{forecast,2}=t_{current,2}+h$, where $t_{forecast,1} \neq t_{forecast,2}$), the distributions coincide.
If $\mathcal{H}$ contains only A2$'$, and A1 relates to $\mathcal{A}$, i.e., it is assumed that the LLM correctly cuts off information beyond the cutoff date, this test is a proxy for context bias. If $\mathcal{H}=\{A1, A2'\}$, then the test is a proxy for the presence of at least one of the two biases. If Fake date test I is passed, we choose the first partition, since the hypothesis that A1 holds is not rejected; the test is then formulated as follows:
Proposition 2 (Fake date test II). When $\mathcal{A} = \{A1\}$ and $\mathcal{H} = \{A2'\}$ are satisfied, the distributions of forecasts $y^1_{forecast}$ and $y^2_{forecast}$ with $d=0$ must coincide.
Proof. See Appendix (ref).
However, if Fake date test I fails, A1 cannot be assumed to hold; therefore, in this case Fake date test II is formulated as follows:
Proposition 2$'$ (Fake date test II). When $\mathcal{A} = \{\}$ and $\mathcal{H} = \{A1,A2'\}$ are satisfied, the distributions of forecasts $y^1_{forecast}$ and $y^2_{forecast}$ with $d=0$ must coincide.
The fake date tests proposed above are formal, but they can be interpreted in different ways. We interpret Fake date test I as a check for lookahead bias, and Fake date test II as a check of how the model uses knowledge near the cutoff boundary. This section discusses why we interpret these tests in this way, and what pitfalls may arise in this.
Passing these tests serves precisely as another advantage of the LLM not containing the corresponding biases, but is not the ground truth. At the same time, failing the tests does not necessarily mean the presence of the corresponding biases, but serves as a serious signal to question the validity of the methodology used to evaluate the model's retrospective accuracy and rank it relative to other models.
If the hypothesis of the equality of distributions is not rejected in Fake date test I, then even with A2 satisfied, this does not mean that the model does not use information outside the cutoff boundary. It may happen that in both forecasts ((ref)) and ((ref)), the LLM uses data outside the cutoff boundary while leading to the same forecasts. One example of such behavior could be a situation where the LLM cuts off data not after the cutoff boundary, but slightly later, and A2 is satisfied with the availability of this information set. In this case, the forecast distributions will also coincide, but information from the future will leak into both forecasts. One can come up with a number of more complex potential situations where the hypothesis that the distributions coincide will not be rejected despite information leakage. For example, if the LLM with a forecast date preceding the model release date cuts off information correctly, while failing to do so when the forecast date is beyond the release date (or vice versa). At the same time, the LLM uses this information differently to build a forecast, obtaining identical distributions. We do not claim to provide a complete and comprehensive description of all such cases, but only want to additionally emphasize that the test results should not be interpreted too optimistically and unequivocally.
A variation of the latter example, when the LLM correctly cuts off information with a forecast date within the training sample and incorrectly with a forecast date outside it, can also be used in the reverse situation when Fake date test I is not passed. If the LLM uses the available information equally, but there is more information for a fake date, the forecasts must not coincide. At the same time, there is no lookahead bias for in-sample forecasts. A second example could be a situation where, for both forecast dates, the LLM does not use information outside the cutoff boundary, but for different dates inside this boundary the model looks at different information, i.e., it is sensitive to the forecast date. In this case, the test may also fail despite correctly following the instruction to ignore information outside the cutoff boundary. The model may also have a bias toward specific numbers in the prompt. For example, the model may be sensitive to dates under identical conditions and generate answers that differ for even or odd months/quarters/years, or years that most frequently appear in science fiction. Although the last example with science fiction seems highly artificial, it is meant to emphasize that LLMs are black boxes, and therefore, unexpected effects can arise inside them that are very difficult to predict in advance without facing them in a specific case. As in the situation with passing Fake date test I, the list of examples is naturally not exhaustive. Moreover, in both the case of passing the test and the reverse situation, various effects can overlap each other. By giving the examples outlined above, we only want to say that the test results should be interpreted with a degree of caution and try to verify the assumptions on which the test is based whenever possible.
We tried to partially verify Assumption 2 for Fake date test I by choosing a cutoff date that definitely satisfies Assumption 1. Such a date was chosen close to the models' release dates (August 15, 2025). For all models used in Section (ref), we selected statistics for five dates\footnote{November 15, 2008 (a date during the Great Recession when forecasts with real and fake dates differ significantly), August 15, 2013 (a random date from the sample), May 15, 2020 (the beginning of the pandemic), August 15, 2021 (a date when the YoY GDP growth indicator has a low base effect), and February 15, 2025 (the last date in our sample).} and looked at how the dynamics of the forecast distributions behave until 2035. Despite the fact that the p-values for the Kolmogorov-Smirnov permutation test (Tables (ref)-(ref) in Appendix (ref)), which is used when analyzing the results in Section (ref), are slightly shifted toward zero, which should not be the case with ideal fulfillment of Assumption 2, we did not see such significant differences in forecasts, except for a number of situations, as when comparing forecasts with real and fake dates. Although these results do not allow us to conclude that Assumption 2 is fulfilled ideally, its approximate fulfillment seems quite sufficient for Fake date test I in practice (see more detailed discussion in Subsection (ref)). For other prompts and models, the results may differ, so we recommend running such tests anew for each experiment.
It should also be noted that Assumption 2 is quite strict in the sense that it requires not just the convergence of forecasts as the forecast date increases, but their exact matching starting from a certain period. This assumption can be weakened by introducing a certain neighborhood in which forecasts must lie starting from $t_{current} \geq t_{cutoff} + d$. Adjusting statistical tests for the selected neighborhood is not so complex; however, selecting this neighborhood in practice is associated with a number of technical and computational difficulties, and to avoid going into details, we introduce a less realistic but technically simpler assumption\footnote{Assumption 2 can also be adjusted to account for seasonality, which, as shown in Subsection (ref), may be relevant for some models and forecast indicators. This adjustment in practice requires that when forecasting with real and fake dates, only the year differs, but not the day or month of the forecast within a year. As a consequence, adjustment of the statistical test is not required in this case.}
As for Fake date test II, its main assumption is A1, which is checked within Fake date test I. If it is satisfied, arguments similar to those in Fake date test I can also be applied to Fake date test II regarding interpretation. Thus, if the test fails, situations may arise where the mismatch of distributions is the result of the model not using information near the forecast date, but interpreting the mechanisms of how the economy works differently depending on which forecast date is used. In the case of passing, one can come up with a number of examples where several effects overlap in such a way as to compensate for each other. For example, due to the compensation of the latter effect with bias associated with context. Alternatively, the LLM may find a historically similar precedent in the data given in the prompt, which exactly corresponds to the historical forecast date, and use this information to fill in the missing context, which will thus lead to identical forecast distributions.
It is also worth emphasizing that Fake date test I uses different forecast and cutoff dates, which, strictly speaking, is not equivalent to a typical LLM-based macroeconomic forecasting setup, in which these dates coincide. Consequently, beyond the arguments outlined earlier, following (or not following) the instruction to cut off information in Fake date test I may not fully translate into following (or not following) this instruction in a real-world problem and should be considered only as an additional argument supporting the absence (or presence) of lookahead bias. A simple example of this interpretational gap occurs when the model misinterprets the temporal boundaries and implicitly treats the forecast date as the cutoff date. In such cases, a failure to pass Fake date test I stems from misinterpreting the task rather than disregarding the cutoff instruction. In Fake date test II, the forecast and cutoff dates coincide; therefore, this problem is absent. Although Fake date test II cannot disentangle lookahead bias from context bias, it provides an additional argument for or against the validity of using in-sample forecasts to assess LLM accuracy when Fake date test I fails.
Despite the potentially possible effects described above, we tend to interpret the proposed tests as a test for lookahead bias and a test for bias associated with additional knowledge in the vicinity of the forecast date (if Fake date test I does not fail). These explanations seem to us the most likely from the point of view of how models that mimic expert reasoning with broad knowledge about how the world is organized would behave. Nevertheless, we understand that LLMs are just complex statistical models, and such interpretations may contain anthropomorphic bias (see Ludwig et al. Ludwig2025), and we urge caution in respect of their conclusions, using them only as one of the arguments for or against the presence of the corresponding biases.
It is also worth noting that, assuming that our interpretation is correct and Fake date test I is passed, the forecast for a fake date and zero depth (forecast ((ref)) with $d = 0$) can be considered as a candidate for evaluating in-sample accuracy. This forecast, provided that the interpretation is correct and Fake date test I is passed, should not contain lookahead bias and may not contain context bias. However, we leave the study of such possibilities for future research for two reasons. First, none of the models in Section (ref) passes Fake date test I, which does not allow us to obtain such a candidate in practice. Second, a number of additional tests should be developed to test the assumptions that must be satisfied so as to avoid various additional biases, such as bias associated with memorizing precedents in the description of the limitations of Fake date test II.
This section describes the results of our experiments that illustrate the application of the methodology outlined in Section (ref). First, we describe the experimental setup, including the models, data, and prompts used. Then we demonstrate general results based on histograms of forecast distributions. At the end of the section, we show a number of cases that help better understand the results obtained.
For the experiments, we selected three open-weight models: \href{https://huggingface.co/moonshotai/Kimi-K2-Instruct}{Kimi-K2 Instruct} (see Kimi Team Kimi2025), \href{https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507}{Qwen3 Instruct} (see Qwen Team Qwen2025) and \href{https://huggingface.co/deepseek-ai/DeepSeek-V3.1}{DeepSeek-V3.1} (see DeepSeek-AI (DeepSeek2025)). At the start of conducting the experiments, all the three models were among the top-10 of Chatbot Arena (among text models) and leading open-weight non-reasoning models. We choose non-reasoning models because, compared to reasoning models, they allow relatively quick responses, which significantly saves time in experiments. Nevertheless, without any changes, the proposed methodology can be applied to reasoning models as well.
The prompts used in the experiments\footnote{All requests to the models were made through \href{https://bothub.chat}{BotHub}, a provider that allows access to different LLMs via one application programming interface (API).} are similar to those shown in Figure (ref) in Section (ref). As variables for forecasting, we selected the U.S. data for the lower bound of the Fed interest rate, YoY CPI growth, and YoY real GDP growth one year ahead from the last known statistical publication date at the time of forecasting. As statistics that the forecast is based on, we took the latest available values of the said variables (vintages\footnote{The first release of seasonally adjusted data for the CPI and real GDP were taken from \href{https://www.philadelphiafed.org/surveys-and-data/real-time-data-research/real-time-data-set-for-macroeconomists}{The Philadelphia Fed's Real-Time Data Set}.} for the CPI and GDP). Forecasts were built for the period from May 2005 to February 2025 with quarterly frequency (80 forecasts in total). As the forecast date, we took the fifteenth day of the second month of a quarter, as a date close to the time of release of inflation data for the previous month and GDP data for the previous quarter. The fake forecast date was chosen as February 15, 2030, as a date that is sufficiently far from the model release dates. The cutoff boundary for Fake date test I was shifted three years back from the real forecast date. Such a period corresponds to the standard medium-term horizon of macroeconomic forecasts, and with this time horizon, forecasts should be almost independent of the current economic context. Furthermore, the selected period does not contradict the results regarding the stabilization horizon of forecasts in the test of Assumption 2. Figure (ref) shows the prompts used to forecast all three variables.
It should be noted that the set of macroeconomic context variables selected for provision to the LLM is far from exhaustive. However, our goal is not to build the best forecast, but only to illustrate the proposed methodology, for which a specific set of indicators does not play the key role, so we focused solely on the main macroeconomic variables.
For each date, $N=100$ forecasts were sampled with real and fake dates with a temperature equal to 0.7. To compare the sample distributions of forecasts, a permutation test (see Ritzwoller et al. Ritzwoller2025) with the Kolmogorov-Smirnov statistic with 10,000 permutations was used.
Figures (ref)-(ref) show general results of Fake date test I for all three models. The results are presented in the format of bubble histograms with the circle size proportional to the density of the forecast distribution. Red indicates forecasts for a real date, blue shows forecasts for a fake date, yellow denotes the real data corresponding to the vintage available on the forecast date. More illustrative forecast distributions together with the p-values of the permutation test with the Kolmogorov-Smirnov statistic for each model and for each date are presented in Figures (ref)-(ref) in Appendix (ref)\footnote{Results are presented in histogram format with the bin size equal to 0.25 for the lower bound of the Fed interest rate and 0.1 for CPI and real GDP growth, corresponding to the natural forecast step of these indicators. The headers show the p-values of the permutation test with the Kolmogorov-Smirnov statistic.}.
The visual analysis of Figures (ref)-(ref) suggests that the forecast distributions\footnote{For some queries, the models, and especially Qwen3 Instruct, state insufficient data with our prompt and output a forecast of 0.0 or `N/A'. We did not exclude such values from the sample, as their share, as can be seen from the figures, is small.} for some dates significantly differ, which is confirmed by the p-values for the Kolmogorov-Smirnov permutation test\footnote{We also conducted tests with the Wasserstein distance. They do not differ qualitatively from those presented in Appendix (ref), so to save space we do not present them herein.} presented in Appendix (ref). When applying the Bonferroni correction for conservative multiple hypothesis testing, the hypothesis of equal distributions is also rejected for any conventional significance level for all models and all variables\footnote{The maximum p-value among all models and variables was 0.008 for 10,000 permutations, due to the p-value discretization level of 0.0001 for individual hypothesis testing. To obtain less discretized p-values in a computationally efficient manner, we estimated the p-value for 1,000,000 permutations for the forecast based on statistics as of November 15, 2008. Since the Bonferroni correction for multiple hypothesis testing depends only on the smallest p-value, this value can be used as the estimate of the upper bound for the p-value of the multiple hypothesis. The maximum value in this case was 0.00008.}. This is a signal that the Kimi-K2 Instruct, Qwen3 Instruct, and DeepSeek-V3.1 models might potentially fail to effectively follow the instruction to cut off information, which is a reason to doubt the correctness of the procedure for evaluating the quality of these LLMs by calculating in-sample metrics in the proposed macroeconomic forecasting task.
The differences in forecasts vary from date to date and from model to model. However, it can be noted that these differences become especially noticeable around the 2007-2009 financial crisis and the 2020 pandemic. Also, forecasts significantly differ on some dates of significant changes in macroeconomic variables outside these periods. This can be seen, for example, in forecasts of the lower bound of the Fed interest rate in 2010-2012 when CPI growth (YoY) and real GDP growth (YoY) left negative territory. The forecasts for a real date in this period are almost always in the vicinity of zero, while the forecasts for a fake date are often higher. Another example could be a decrease in forecasts of real GDP growth (YoY) for a fake date when CPI growth (YoY) decreased in 2015.
Since none of the LLMs considered has passed Fake date test I, we conduct Fake date test II as a proxy test for the presence of at least one of the two biases, not only context bias. Figures (ref)-(ref) show bubble histograms similar to those in Figures (ref)-(ref). To save space, we omit figures similar to those given in Appendix (ref). The p-values of the tests are provided in Tables (ref)-(ref) in Appendix (ref). As with Fake date test I, the distributions of forecasts for real and fake dates differ markedly for many time periods, exhibiting patterns similar to those observed in Fake Date Test I. The test with the Bonferroni correction also rejects the hypothesis of equality of distributions at all conventional significance levels. As noted in Section (ref), this result, in addition to Fake date test I, serves as another signal of the unsuitability of in-sample metrics for evaluating the forecast quality of the Kimi-K2 Instruct, Qwen3 Instruct, and DeepSeek-V3.1 models in the forecasting task considered herein.
To better understand the results, we will consider several examples that help shed some light on the nature of the demonstrated forecast distributions.
The first thing we would like to focus on is a more detailed analysis of forecast distributions within Assumption 2. As noted in Section (ref), this assumption is not perfectly satisfied. Tables (ref)-(ref) in Appendix (ref) present the p-values for the Kolmogorov-Smirnov permutation test. In the case of ideal fulfillment of Assumption 2, conservative p-values (see Ritzwoller et al. Ritzwoller2025) for the permutation test should be either distributed uniformly or partially shifted toward one. The values in Tables (ref)-(ref) have a higher density in the neighborhood of zero\footnote{It should be noted that individual p-values in one column are not independent due to the common forecast as of February 15, 2030, and therefore the analysis was conducted for all dates and tables.}. Nevertheless, for most forecast dates, individual p-values are generally adequate (far from zero or not extremely close to zero), which allows us to speak of approximate fulfillment of Assumption 2.
To illustrate how sensitive the results obtained in the previous subsection are to the non-fulfillment of Assumption 2 in the exact form, we selected several triples --- a set of statistical data, a model, and a forecasting indicator --- for which the deviations are the largest. For this, we excluded the first three years of forecasting from the analysis to avoid potential biases associated with the presence of context, i.e., only dates starting from August 15, 2028 were taken. Then, we selected the triples for which the p-values from Tables (ref)-(ref) do not exceed the minimum threshold of 0.001 in at least 10 percent of cases.
For the Kimi-K2 Instruct model, no such pairs were found. In the case of the DeepSeek-V3.1 model, there are three such situations: the forecast of the lower bound of the Fed interest rate and the forecast of real GDP growth based on statistics as of November 15, 2008, the forecast of the lower bound of the Fed interest rate based on statistics as of August 15, 2021. As can be seen from Figure (ref), which shows forecasts for real and fake dates, as well as two forecasts within the testing of Assumption 2 with the maximum Wasserstein distance (as a distance having interpretation in terms of the transportation problem and being an upper bound to the difference of means of distributions), the sensitivity of the model to the forecast date when unable to look ahead is significantly lower than the difference of forecasts with real and fake dates, which indirectly confirms the weak influence of pure sensitivity to the forecast date on the results of Subsection (ref) in the DeepSeek-V3.1 model.
For the Qwen3 Instruct model, seven such pairs were found. These are all forecasts of real GDP growth, as well as forecasts of CPI growth based on statistics as of August 15, 2021 and February 15, 2025. Figure (ref) shows that the cases of forecasts of real GDP growth based on statistics as of November 15, 2008 and May 15, 2020 and of CPI growth based on statistics as of August 15, 2021 are similar to those obtained by the DeepSeek-V.3.1 model, and the differences are not as great as those for forecasts with real and fake dates. Moreover, the main divergence in CPI growth forecasts is due to a large number of model refusals to provide a forecast. Excluding such points significantly reduces the Wasserstein distance between the forecast distributions within the testing of Assumption 2. Forecasts with statistics as of August 15, 2013 and August 15, 2021, as we saw from the additional analysis, have pronounced seasonality. When comparing the forecast distributions not against February 15, 2030, but against the corresponding month of 2030, the p-values of the tests similar to those presented in Tables (ref)-(ref) satisfy the selected threshold. This result suggests that choosing a single date when conducting Fake date test I may not be the best solution in some cases and casts doubt on the test results for real GDP growth for Qwen3 Instruct in Subsection (ref) for May, August, and November. However, as shown in Figure (ref) and Figures (ref)-(ref) in Appendix (ref), a significant difference in forecasts when applying Fake date test I remains even when comparing forecasts for a real date against the corresponding month of 2030 as a fake date.
Forecasts of CPI growth and real GDP growth based on statistics as of February 15, 2025 also show sensitivity to the date, which is comparable in magnitude to the difference in forecasts with real and fake dates. For real GDP growth, this happens with a small but statistically significant difference in the metric for comparing the forecast distributions with real and fake dates. In such cases, sensitivity to the date can indeed become a reason for additional false rejection of the hypothesis of the equality of distributions, but as can be seen from the figures presented in the previous subsection and figures in Appendix (ref), the forecast distributions differ quite strongly for a large number of dates and, although potentially influencing part of the results, this effect cannot fully explain the difference in the results of Fake date test I.
To better understand the reasons for large discrepancies in forecasts with real and fake dates in Fake date test I for some periods, we selected two dates: November 15, 2008 and May 15, 2021. For each of the previously used LLMs, each forecast indicator, and both dates, we randomly selected 20 responses with real and fake dates and analyzed them manually.
For the forecast based on statistics available as of November 15, 2008, all models for all indicators give forecasts that for a real date are lower than for a fake date. The main reason for this is how the models interpret the statistics on the lower bound of the Fed interest rate, CPI growth, and real GDP growth. When forecasting for a real date, these data are interpreted as the beginning of a recession in most cases. Kimi-K2 Instruct in the most obvious way does not follow the instruction not to use data after November 15, 2005, often referring to indicators and events between the cutoff date and the forecast date, such as the rate-cutting cycle, financial system stress, and trends in commodity market price dynamics (examples of model responses are provided in Appendix (ref)). DeepSeek-V3.1, on the other hand, does not always directly reference real events of this period, but nevertheless concludes that the economy is either entering or already is in a recession phase. One of the arguments that recurred in the responses in favor of the fact that the risks of a decline in GDP (real GDP growth of 0.8% (YoY)) dominate over inflation risks (CPI growth of 3.7% (YoY)) is a historically low level of the lower bound of the Fed interest rate (1 percentage point). This seems to be a very logical conclusion, and the forecast based on the model's explanation is often very difficult to suspect of “looking” beyond the cutoff boundary. However, the comparison with identical forecasts for a fake date, where DeepSeek-V3.1 along with other LLMs, even with low rates, evaluate the risks of rising inflation as more important for the economy, makes one doubt that the forecast for a real date does not use additional information contained in the LLM weights, even if this is in no way reflected in the explanation of the obtained forecasts.
The analysis of forecasts based on statistics as of May 15, 2021 shows similar patterns. The data provided are largely similar to the information available in November 2008: high inflation (CPI growth of 4.2% (YoY)), low growth (real GDP growth of 0.4% (YoY)) and interest rate (the lower bound of the Fed interest rate equal to 0 percentage points). The indicator we would like to focus on is real GDP growth. Forecasts of this variable for a real date turn out to be higher in all models than forecasts for a fake date. The reason for this is that when forecasting for a real date, all LLMs understand that the economy is in the recovery phase after the COVID-19 shock. All models in one way or another signal this in their explanations of the forecasts given. In some cases, this happens explicitly, as, for example, in many responses of Kimi-K2 Instruct, while in others, as with most responses of Qwen3 Instruct, in the form of information that the model cannot use when producing a forecast due to prompt restrictions related to the unavailability of information after May 15, 2018.
In the analyzed LLM responses for November 15, 2008 and May 15, 2021, we did not detect, except for a few cases, any direct references to data and events beyond the forecast date; however, we are not inclined to interpret this as the models not using this information in their responses in any way. This can happen implicitly (see Arcuschin et al. Arcuschin2025, Chen et al. Chen2025Anthropic), through the use of model weights where information about these periods is stored, without explicit indication in the tokens generated for explanation due to the direct restriction in the prompt. This question requires additional study and potentially more advanced and computationally complex tools\footnote{As far as we know, the question of systematically detecting a discrepancy between model explanations and its “true motives" is open in the literature. However, new techniques, such as, for example, interpretation based on sparse autoencoders or transcoders (see, for example, research by Ameisen et al. Ameisen2025 and Lindsey et al. Lindsey2025), allow increasingly deep penetration into the “decision-making” processes of LLMs (see Marks et al. Marks2025).}, which is beyond the scope of this study. Nevertheless, the analysis results show that all models cannot accurately follow the instruction not to use information beyond the cutoff boundary, which, together with the results of Fake date test II, is a signal regarding the potential incorrectness of using the procedures for evaluating the accuracy of macroeconomic forecasts of the considered LLMs based on in-sample metrics (at least for the prompt used in this paper and the set of macroeconomic statistics provided to the models).
Neither did we see in the analyzed responses, except for a few cases, any references to historical data or precedents beyond the information cutoff date in forecasts with the fake date\footnote{In addition to the analysis described above, we conducted a focused analysis of the occurrence of the character strings `pandem' and `covid' in responses generated with fake dates for both tests. We found that such mentions were nearly absent in responses dated prior to May 15, 2020. Moreover, almost all earlier occurrences were associated with explicit statements regarding the impossibility of utilizing pandemic-related information for subsequent forecasting.} \footnote{It should be noted that in a number of preliminary experiments with a softer prompt regarding data cutoff, we noticed mentions of historical precedents beyond the cutoff boundary in forecasts for a fake date.}, which, in addition to the candidate with $d=0$ discussed earlier in Section (ref), makes these forecasts a potential candidate for using in in-sample testing; however, as in the case of using data beyond the forecast date with the real date, this can happen implicitly. A more detailed study of the properties of such candidates requires separate research.
The final point we would like to address in this subsection is a brief analysis of LLM responses within Fake date test II. This test checks for the presence of at least one of two biases: lookahead bias and context bias. Due to the absence of a restriction on the use of knowledge for the past three years, models explicitly incorporate information about ongoing economic processes when forecasting for a real date. For example, when forecasting based on statistics as of May 15, 2021, all models --- in contrast to the forecast generated for a fake date --- refer to the COVID-19 shock of 2020 (see Appendix (ref)), which is a sign of context bias in the in-sample forecasts. Furthermore, the use of information beyond the forecast date is particularly difficult to detect from model explanations; however, as noted earlier, such leakage may occur implicitly. A comprehensive analysis of the explanations given by the three models tested in this study to identify all such cases is beyond the scope of this paper. Nevertheless, a selective manual review reveals that such examples are present in the dataset, signaling not only context bias but also a model's failure to follow the instruction to truncate information beyond the cutoff date. As an example of such behavior, Figure (ref) shows the Qwen3 Instruct model's response for the inflation forecast based on statistics as of August 15, 2008. This date falls at the start of the oil price decline from the July 2008 peaks. Although the decline had only just begun and the subsequent trajectory was not obvious from futures prices or analytical reports of the time, the model's explanations refer to the sharp drop in oil prices in late 2008 and early 2009, demonstrating that the model has explicitly used of future information when generating the forecast.
In this paper, we propose tests designed to shed at least some light on the possibility of using in-sample forecasts of LLMs to evaluate the quality of their forecasts. The fake date tests are positioned as an applied tool that gives signals regarding the presence of lookahead bias and bias associated with an additional context beyond that specified in the prompt, and consequently are a reason to question the validity of using the procedure for evaluating the accuracy based on in-sample forecasts if the tests are not passed. Although this study focuses on macroeconomic forecasts, the proposed tests can be applied to a wider range of tasks, which includes, at a minimum, tasks of time series forecasting.
The biases discussed in this study may potentially be relevant in a number of other tasks that use LLMs. Such tasks include, for example, studies related to sentiment-based forecasting, modeling of survey data or human decision-making. Evaluation of the predictive ability of news, inflation expectations based on LLM responses, and agent-based models, where agents are modeled not based on simple predetermined rules or reinforcement learning, but based on LLMs, can serve as examples of such tasks in economics. How significant the biases studied herein are in these tasks is a separate and rather complex question that we do not investigate, but we believe that with some modifications and, possibly, optimization of the number of requests, the proposed tests can help in detecting biases in these directions as well.