The exact contents of citations.db main_text.text for this paper — one flattened LaTeX string, title through conclusion, appendix excluded, unmodified except for removing email addresses. This is what our citation measures are computed over.
98,767 characters
LLM-Generated Counterfactual Stress Scenarios for Portfolio Risk Simulation via Hybrid Prompt-RAG Pipeline
\title{LLM-Generated Counterfactual Stress Scenarios for Portfolio Risk Simulation via Hybrid Prompt-RAG Pipeline}
\author{Masoud Soleimani}
\email{[email removed]}
\orcid{0009-0003-3024-2289}
\affiliation{
\institution{Department of Information Engineering, University of Pisa}
\city{Pisa}
\country{Italy}
}
\begin{abstract}
We develop a transparent and fully auditable LLM-based pipeline for macro–financial stress testing, combining structured prompting with optional retrieval of country fundamentals and news. The system generates machine-readable macroeconomic scenarios for the G7, which cover GDP growth, inflation, and policy rates, which are translated into portfolio losses through a factor-based mapping that enables Value-at-Risk and Expected Shortfall assessment relative to classical econometric baselines. Across models, countries, and retrieval settings, the LLMs produce coherent and country-specific stress narratives, yielding stable tail-risk amplification with limited sensitivity to retrieval choices. Comprehensive plausibility checks, scenario diagnostics, and ANOVA-based variance decomposition show that risk variation is driven primarily by portfolio composition and prompt design rather than by the retrieval mechanism. The pipeline incorporates snapshotting, deterministic modes, and hash-verified artifacts to ensure reproducibility and auditability. Overall, the results demonstrate that LLM-generated macro scenarios, when paired with transparent structure and rigorous validation, can provide a scalable and interpretable complement to traditional stress-testing frameworks.
\end{abstract}
\begin{CCSXML}
<ccs2012>
<concept>
<concept_id>10010147.10010178.10010187</concept_id>
<concept_desc>Computing methodologies~Natural language generation</concept_desc>
<concept_significance>500</concept_significance>
</concept>
<concept>
<concept_id>10002951.10003260.10003309</concept_id>
<concept_desc>Information systems~Retrieval models and ranking</concept_desc>
<concept_significance>400</concept_significance>
</concept>
<concept>
<concept_id>10010405.10003550.10003555</concept_id>
<concept_desc>Applied computing~Economics</concept_desc>
<concept_significance>300</concept_significance>
</concept>
<concept>
<concept_id>10002944.10011123.10011130</concept_id>
<concept_desc>General and reference~Evaluation</concept_desc>
<concept_significance>200</concept_significance>
</concept>
</ccs2012>
\end{CCSXML}
\ccsdesc[500]{Computing methodologies~Natural language generation}
\ccsdesc[400]{Information systems~Retrieval models and ranking}
\ccsdesc[300]{Applied computing~Economics}
\ccsdesc[200]{General and reference~Evaluation}
\keywords{Large Language Models, Retrieval-Augmented Generation, Financial Stress Testing, Tail Risk, Scenario Generation}
\maketitle
\section{Introduction}
Macroeconomic stress testing is central to financial stability analysis and bank supervision \cite{Taskinsoy2022Stress}. Stress tests articulate adverse but plausible macroeconomic conditions to assess vulnerabilities, determine capital adequacy, and inform policy design. Regulatory authorities such as the Federal Reserve and the ECB provide top–down narratives, while financial institutions implement internal frameworks based on historical replays, econometric models, or Monte Carlo simulations \cite{thiemann2021measuring, Maleki2024HealthcareVideo}. Yet, despite their institutional importance, traditional stress–testing pipelines face persistent challenges. First, they struggle to represent low-frequency disruptions such as pandemics, supply-chain failures, energy shocks, or geopolitical fragmentation \cite{Borio2014StressTesting, Carney2015Horizon} that fall outside econometric training windows. Second, scenario design remains manually intensive \cite{Baer2023ClimateScenarios, Aikman2024ECBStress} and difficult to scale across jurisdictions or portfolios. Third, econometric systems are often slow to adapt to real–time information \cite{Pesaran2005RealTime}, limiting responsiveness in fast-moving macro–financial environments.
\medskip
\noindent\textbf{Large Language Models (LLMs)} offer a promising complement. Their ability to synthesize structured macro narratives from heterogeneous information sources has been demonstrated across domains including software engineering \cite{hou2024large, kiashemshaki2025secure, nam2024using, torkamani2025streamlining}, education \cite{wen2024ai, Chu2025LlmAgents}, and policy analysis \cite{safaei2024end, chen2025clear, Gapud25}. For stress testing, LLMs can rapidly generate country-specific macroeconomic scenarios while remaining interpretable to human analysts. However, unconstrained generation poses well-known risks: hallucination, numerical drift, internal inconsistency, and limited reproducibility \cite{ji2023towards, huntsman2024prospects, staudinger2024reproducibility, Tan2025ConsistentLLMs, Tonmoy2024HallucinationSurvey}. These issues motivate hybrid architectures with explicit grounding, structure, and diagnostics.
\medskip
\noindent In this paper we develop and evaluate a transparent, retrieval–optional pipeline for macro–financial scenario generation. Our system couples structured country profiles with optional news retrieval, prompting GPT–5–mini and Llama-3.1-8B-Instruct to emit machine-readable macro shocks (GDP, inflation, interest rates). These shocks are then translated into portfolio losses through a linear factor channel, enabling direct computation of scenario-induced VaR and CVaR multiples relative to historical and econometric baselines. The design preserves narrative flexibility while enforcing numerically stable, auditable outputs.
\paragraph{Motivation.}
Traditional econometric stress tests struggle to scale or update quickly, while fully generative LLM approaches lack governance guarantees. Our aim is to bridge these approaches: retaining the interpretability and auditability of structured stress-testing frameworks while exploiting the adaptability and expressiveness of modern LLMs.
\medskip
\noindent\textbf{The contributions of this paper are:}
\begin{enumerate}
\item A fully auditable Prompt–RAG pipeline for macro–financial scenario generation, with structured JSON outputs and optional grounding via country profiles and news retrieval.
\item A comprehensive G7 experiment comprising \emph{840 intended scenarios per model}
(7 countries $\times$ 30 prompt variants $\times$ 4 retrieval configurations),
of which 627/617/307 survive plausibility filtering for deterministic GPT-5-mini,
non-deterministic GPT-5-mini, and Llama-3.1-8B-Instruct, respectively.
\item A consistent macro$\rightarrow$portfolio mapping using a linear factor channel, enabling computation of VaR/CVaR multiples relative to historical bootstrap, EWMA, and GARCH(1,1)–t baselines.
\item Extensive diagnostics including scenario plausibility checks, dispersion analysis, cross-run stability, fairness metrics, and ANOVA variance decomposition, showing that portfolio composition and prompt design dominate risk variance, while RAG/news have only marginal effects.
\item A complete reproducibility and governance layer: deterministic run modes, hash-verified artifact manifests, and explicit snapshotting to ensure replayability across models and retrieval configurations.
\end{enumerate}
From a practitioner perspective, the pipeline is best viewed as a
``scenario generator'' for risk committees: given a fixed regulatory or
internal baseline, the LLM produces a menu of country-specific,
narrative-rich shocks that can be screened, edited, and selectively
added to an institution's scenario library, rather than replacing
existing frameworks.
\section{Related Work}
\label{sec:related}
\paragraph{Large language models in finance.}
Early NLP applications used domain lexicons and linear models, but struggled with contextual nuance.~\cite{mishev2020evaluation, Tetlock2007InvestorSentiment, LoughranMcDonald2011Liability}.
Transformer architectures and later, large language models (LLMs), closed that gap \cite{Araci2019FinBERT, Yang2020FinBERT}. The survey of Xing et al.\ documented over 40\% accuracy gains relative to bag-of-words baselines on stock-return prediction tasks~\cite{xing2018natural}.
Domain-specific pre-training further boosts performance: BloombergGPT (50B parameters) outperformed general models by up to 15 pp on 14 financial NLP benchmarks~\cite{wu2023bloomberggpt}, while the open-source FinGPT project emphasises continual web-scale fine-tuning for reproducibility~\cite{liu2023fingpt}.
LLMs also show strong zero-shot capabilities: ChatGPT improves short-horizon equity-return forecasts from headlines~\cite{LopezLira2024PredictiveEdge}, and GPT-4 can generate analyst-style reports that meaningfully influence professional investors~\cite{takayanagi2025can, Gruver2023ZeroShotTS, Carriero2024MacroLLM}.
\paragraph{Machine-learning stress testing.}
Traditional macro–financial stress tests rely on vector autoregressions and hand-crafted macroeconomic shocks \cite{Guarda2012MVAR, Aikman2024ECBStress, AlfaroDrehmann2009MacroStress}.
Machine-learning variants broaden the factor set and automate aspects of scenario design \cite{FlaigJunike2022ScenarioGen, Petropoulos2022DeepStress}.
Packham shows that PCA and autoencoders uncover latent factors that better explain tail losses than regulator-supplied shocks~\cite{packham2024risk}.
Bueff et al.\ propose counterfactual generators for credit portfolios that yield interpretable “closest-possible’’ stress scenarios~\cite{bueff2022machine}.
Moffo demonstrates that gradient-boosted trees outperform linear satellite models in the CCAR environment, while noting their limited narrative interpretability~\cite{moffo2024machine}.
These approaches operate on structured numerical data and do not generate full narrative macroeconomic scenarios, despite the growing emphasis on explainable stress testing \cite{Nallakaruppan2024XAI, Sun2025ExplainableEnsemble}.
\paragraph{Retrieval-augmented generation (RAG)}
LLMs hallucinate when extrapolating beyond their training distribution.
Retrieval-augmented generation (RAG) mitigates this by conditioning model outputs on retrieved ground-truth documents~\cite{lewis2020retrieval, Asai2024SelfRAG, Yan2024CorrectiveRAG}.
WebGPT augmented GPT-3 with a browser and citation mechanism, raising factual accuracy to 74\% on long-form QA~\cite{nakano2021webgpt}.
Atlas matched GPT-3-175B performance using an 11B model paired with a neural retriever and frozen language model~\cite{izacard2023atlas}.
In finance, Zhang et al.\ integrate a news-article retriever with GPT to improve sentiment accuracy by up to 48\% over non-retrieval baselines~\cite{zhang2023enhancing}.
To our knowledge, no prior work combines RAG with full macroeconomic stress-scenario generation and downstream portfolio risk translation.
\paragraph{Position of this study.}
This work bridges these strands.
We combine IMF fundamentals with optional news retrieval using a MiniLM--FAISS
retriever to ground prompts for GPT-5-mini and Llama-3.1-8B-Instruct.
The hybrid Prompt--RAG pipeline produces structured JSON macro shocks that
(i) are statistically plausible relative to historical and econometric
VaR/CVaR baselines, (ii) exhibit cross-country heterogeneity, and
(iii) remain interpretable for supervisory-style review.
To our knowledge, the literature has not yet documented an end-to-end pipeline
that couples retrieval-augmented LLM macro scenarios with explicit
portfolio-level VaR/CVaR translation. We do not benchmark directly against
CCAR/EBA or bank internal scenario libraries; instead, we position our
framework as a complement to those tools, providing a scalable way to
generate additional, grounded narratives around existing macro baselines.
Our factorial G7 evaluation quantifies how models, retrieval choices, and
macro narratives influence tail-risk outcomes.
\section{Methodology}
\label{sec:methodology}
We develop a retrieval-augmented LLM pipeline for generating structured macroeconomic
stress scenarios and translating them into portfolio tail-risk metrics.
The system consists of five stages:
(1) macro and market data ingestion,
(2) document embedding and indexing for retrieval,
(3) structured scenario generation by LLMs with plausibility filtering and regime tagging,
(4) construction of deterministic, LLM-free baselines and regime-specific covariance
matrices, and
(5) portfolio-level VaR/CVaR evaluation via three stress channels
(pure volatility, linear factor, and nonlinear factor).
Figure~\ref{fig:pipeline} presents an overview.
Market factor extraction uses standard PCA methodology \cite{JolliffeCadima2016PCA, ConnorKorajczyk1986APT}, while historical volatility baselines follow RiskMetrics-style EWMA estimation \cite{JPMorgan1996RiskMetrics} and GARCH(1,1)–t models \cite{Engle1982ARCH, Bollerslev1986GARCH}.
\begin{figure*}[t]
\centering
\includegraphics[width=\linewidth]{pipeline_overview}
\caption{Pipeline for scenario generation and risk translation.
IMF fundamentals and optional news are embedded in MiniLM \cite{wang2020minilm} and retrieved via FAISS \cite{douze2025faiss}
to condition the prompt. LLMs output structured JSON scenarios (GDP, inflation, interest rates, rationale,
and sector-level exposures), which are screened by hard and soft plausibility gates
and tagged with a regime label.
Scenario shocks are mapped to asset returns through three channels:
(i) a pure volatility channel that scales the covariance matrix,
(ii) a linear PCA factor channel, and
(iii) a nonlinear polynomial factor channel that contains all text/RAG/news
amplification.
Regime severity $\lambda$ mixes calm and crisis covariance matrices, and all
channels are benchmarked against deterministic, LLM-free baselines.
Ablations toggle model type (GPT-5-mini vs.\ Llama-3.1-8B-Instruct),
retrieval (RAG on/off), and news augmentation (on/off).}
\label{fig:pipeline}
\end{figure*}
\subsection{Pipeline Pseudocode (simplified)}
\begin{lstlisting}
# 1. Macro and news profiles
for country in G7:
base = load_IMF_WEO(country)
news = fetch_recent_news(country) if USE_NEWS else []
profile = build_profile(base, news)
embed = MiniLM(profile)
faiss.add(country, embed)
# 2. Market data, PCA factors, and baselines (LLM-free)
prices = load_cached_prices(assets=ETF_UNIVERSE) # SPY, IEF, GLD, sectors
pca_factors = fit_PCA_factors(prices["SPY","IEF","GLD"],
seed=SEED, sign_align=True)
betas_linear, betas_poly = estimate_factor_betas(prices, pca_factors)
Sigma_calm = estimate_covariance(prices, period="2012-2019")
Sigma_crisis = estimate_covariance(prices, period="GFC+COVID")
baselines = build_deterministic_baselines(WEO, prices, pca_factors)
# 3. Scenario generation across configurations
for cfg in grid(model ∈ {GPT-5-mini, Llama-3.1-8B-Instruct},
RAG ∈ {0,1}, NEWS ∈ {0,1}):
for prompt_variant in PROMPTS: # 30 prompt variants
ctx = retrieve(country, k=3) if RAG else [profile]
out = LLM_generate(model, ctx, template="Q4_2026")
json = extract_JSON(out)
if not passes_hard_plausibility(json):
continue
regime = NLI_classify_regime(json.rationale)
lambda_severity = build_lambda(json, regime_score=regime.score)
if not passes_soft_plausibility(json, lambda_severity):
continue
scenarios.append((json, lambda_severity, cfg))
# 4. Scenario-specific covariance and three stress channels
for s, lambda_severity, cfg in scenarios:
Sigma_scen = (1 - lambda_severity)*Sigma_calm + \
lambda_severity*Sigma_crisis
shocks_macro = macro_shock_vector(s)
shocks_factors = macro_to_PCA(shocks_macro)
mu_linear = linear_drift(betas_linear, shocks_factors, decay=True)
mu_nonlinear = nonlinear_drift(betas_poly, shocks_factors,
lambda_severity, cfg)
# volatility-only channel: zero mean, scaled Sigma_scen
paths_vol = simulate_paths(mu=0, Sigma=scale_cov(Sigma_scen, shocks_macro))
# linear channel: mu_linear, Sigma_scen
paths_lin = simulate_paths(mu=mu_linear, Sigma=Sigma_scen)
# nonlinear channel: mu_linear + mu_nonlinear, Sigma_scen
paths_nonlin = simulate_paths(mu=mu_linear+mu_nonlinear, Sigma=Sigma_scen)
compute_VaR_CVaR_MDD(s, cfg, paths_vol, paths_lin, paths_nonlin, baselines)
\end{lstlisting}
\vspace{0.5em}
\subsection{Macroeconomic Inputs}
Country-level fundamentals come from the April~2025 \emph{IMF World Economic Outlook} (WEO) \cite{IMF2025WEOdata}.
For each G7 economy we extract the latest projections for real GDP growth,
headline inflation, and the short-term interest rate.
These values serve as the baseline from which the LLM generates
counterfactual Q4~2026 stress shocks.
The horizon is fixed across all configurations to ensure comparability.
In addition to LLM-generated scenarios, we construct deterministic macro stress
benchmarks.
First, we implement a fixed ``2008--09-style'' global risk-off shock applied
uniformly across all countries:
$\Delta \text{GDP} = -3$~pp, $\Delta \text{inflation} = +1$~pp,
and $\Delta \text{policy rate} = +1$~pp.
Second, using country-specific WEO projections, we apply a uniform adjustment
of $\Delta \text{GDP} = -5$~pp, $\Delta \text{inflation} = +2$~pp, and
$\Delta \text{policy rate} = +1.5$~pp, yielding reproducible and
cross-country-comparable deterministic shocks that respect the baseline
heterogeneity in the WEO paths.
\subsection{Optional News Integration}
To capture high-frequency macro developments not yet reflected in the WEO,
the system optionally augments each country profile with recent
English-language news headlines. For each G7 country we query a news API
over a fixed lookback window anchored at the run timestamp, using a
country-specific boolean query on macro terms (e.g.\ “economy”, “GDP”,
“inflation”). If the provider rejects the requested window as being “too far
in the past”, the earliest admissible date is parsed from the error message
and the window is clamped forward to the maximum span allowed by the plan.
All retrieved articles are normalised to a fixed-size headline snapshot:
we sort deterministically by \texttt{publishedAt} and title, deduplicate on
title, and then truncate or pad to \textbf{exactly 50 rows per country}. When
fewer than 50 real headlines are available, the remaining rows are filled
with explicit \texttt{[PAD-XX] No headline available} markers so that every
\texttt{\_headlines.csv} file has the same schema and row count. These
50-row snapshots are saved to disk together with a JSON sidecar containing
the effective time window, query string, and retrieval attempts, and their
SHA-256 hashes are recorded in the run manifest.
For prompt construction we do not pass all 50 headlines directly. Instead,
we embed the real (non-padded) titles with MiniLM-L6-v2 and run
$k$-means clustering with $k=20$, selecting one exemplar per cluster. This
yields up to \textbf{20 diverse headlines per country}, which are inserted
as a “Top-20 diverse headlines” block in the context. If fewer than
20 real headlines exist, we simply return all of them. Padded rows are
never used in the prompt.
In this study, news headlines are \emph{snapshot-pinned}: the 50-row
headline CSV and its JSON metadata are fetched once per country, written
to disk, and treated as immutable. All reported results are computed using
these frozen headline snapshots; subsequent analyses never re-query the
news API.
\subsection{Knowledge Store Construction}
Each country profile (baseline WEO data, with or without news) is serialized to
plain text and embedded using MiniLM-L6-v2.
Embeddings are indexed with FAISS using flat
inner-product search. The retrieval index is deterministic: the input corpus, ordering, and random seed are fixed so that top-$k$ neighbours are reproducible.
Raw documents are hashed and cached for auditability. To guarantee replayability, all inputs to retrieval are snapshot-frozen. This includes: (i) serialized WEO baseline files, (ii) headline CSVs, (iii) MiniLM embeddings, and (iv) the FAISS index itself. All files are hashed and recorded in a run manifest. Retrieval is therefore deterministic conditional on these frozen artefacts.
\subsection{Semantic Retrieval}
For a given query country, the retriever returns the top-$k = 3$ most relevant
country profiles by cosine similarity.
These tend to be economically similar peers (e.g.\ U.S.\ retrieves Canada and the U.K.)
and provide contextual anchors for the LLM.
If RAG is disabled, the prompt includes only the target country's profile.
Retrieval depth $k$ balances contextual richness with token-length constraints.
\subsection{Prompt Construction}
Prompts consist of
(i) a system instruction positioning the LLM as a macro–financial analyst,
(ii) a context block with retrieved WEO fundamentals (and optionally news), and
(iii) a directive to generate a severe but plausible macroeconomic scenario for Q4~2026.
The model must return a structured JSON object with:
\texttt{country}, \texttt{title},
\texttt{gdp\_growth}, \texttt{inflation}, \texttt{interest\_rate},
\texttt{rationale}, and \texttt{risk\_sectors}.
A tolerant parser extracts the first valid JSON object.
\subsection{Language Model Inference}
We evaluate two compact instruction–tuned models:
\begin{itemize}
\item \textbf{GPT-5-mini} (OpenAI, 2025),
\item \textbf{Llama-3.1-8B-Instruct} (Meta, 2025).
\end{itemize}
The full experimental grid (countries, model family, retrieval flag, news flag, and
prompt variants) is described in Section~\ref{sec:experimental}. For each
(country, model, retrieval, news) configuration we generate \emph{30 distinct
prompt variants}, and we produce \emph{one scenario per prompt variant}. Thus each
configuration yields exactly 30 raw scenarios before plausibility filtering.
\textbf{Determinism note.}
Temperature is fixed at near zero, but strict determinism cannot be guaranteed \cite{staudinger2024reproducibility}.
Residual nondeterminism may arise from (i) tokenization or backend differences on
provider-side inference infrastructure, and (ii) retrieval ordering when cosine
similarities between documents are nearly tied. Deterministic behaviour holds only
\emph{conditional on a fixed, snapshot-frozen context block}, including the embedded
WEO baseline, MiniLM embeddings, FAISS index, and frozen news headline snapshots.
\textbf{Plausibility audit and regime tagging.}
Each generated scenario is screened by a two-layer plausibility filter.
A hard gate rejects any scenario with implausible or internally contradictory
macro values (e.g., $|\Delta \text{GDP}| > 10$~pp, inflation $>20\%$,
rates $>15\%$ or $< -1\%$, or deep recessions paired with disinflation and
rate hikes without credible rationale) \cite{hardy2014stress}.
A soft score in $[0,5]$ then evaluates macro magnitude, cross-variable coherence,
and rationale structure, penalising outliers \cite{hopper2022designing}.
In parallel, we classify the free-text rationale using a DeBERTa-based NLI model \cite{he2021debertav3}
into \emph{normal}, \emph{stress}, or \emph{crisis} and obtain a continuous regime
score in $[0,1]$.
Scenarios that fail the hard gate or fall below a minimum soft score are dropped.
For retained scenarios, the macro shocks and regime score are combined into a
scalar severity index $\lambda \in [0,1]$ used later for covariance mixing and
nonlinear drift amplification.
GPT-5-mini exhibits consistently high pass rates across all configurations.
Llama-3.1-8B-Instruct shows concentrated failures primarily in one
retrieval-enabled, news-enabled configuration, with all other configurations
achieving high plausibility retention.
\subsection{Ablation Dimensions}
Throughout the paper we vary three binary design choices in addition to country:
\begin{itemize}
\item \textbf{Model family} (GPT-5-mini vs.\ Llama-3.1-8B-Instruct),
\item \textbf{Retrieval (RAG)} enabled vs.\ disabled,
\item \textbf{News retrieval} enabled vs.\ disabled.
\end{itemize}
Taken together, these switches define the eight system configurations per country that we refer to simply as “configurations” in Sections~\ref{sec:experimental}–\ref{sec:results}, where we study their effects on plausibility, dispersion, tail risk, ANOVA variance decomposition, and fairness diagnostics.
\subsection{Portfolio Stress Mapping (Three-Channel Factor Model)}
Scenario shocks are mapped into portfolio losses using a
\textbf{three-factor PCA model} on ETF returns \cite{FamaFrench1993Factors}, together with
calm/crisis covariance mixtures and three distinct stress channels.
\paragraph{Portfolios.}
We consider two representative portfolios:
\begin{itemize}
\item \textbf{Portfolio A:} U.S.\ equity (SPY), intermediate Treasuries (IEF),
and gold (GLD) with fixed weights of 60/30/10.
\item \textbf{Portfolio B:} an equal-weighted portfolio across the full set of
sector ETFs with sufficient history (XLE, XLF, XLK, XLY, XLI, XLU,
XLV, XLP, XLB, XLRE).
\end{itemize}
Weights are re-normalized daily over a 63-day horizon. We use a 63-day horizon because it corresponds to approximately one trading
quarter, matching the Q4~2026 shock horizon specified in the scenario prompts.
Quarter-ahead propagation is standard in macro–financial stress testing
(e.g., CCAR, ECB, BoE) and provides a stable yet responsive window for PCA
factor estimation: shorter windows (e.g., 21 days) produce noisy loadings,
while longer windows (e.g., 126–252 days) dilute the impact of the intended
macro shock by averaging over overly long historical periods.
\paragraph{PCA factors and betas.}
We compute principal components of the daily excess returns on
(SPY, IEF, GLD) over 2015--2025 with a fixed random seed and
post-hoc sign alignment so that:
$\text{PC}_1$ loads positively on SPY (equity risk),
$\text{PC}_2$ loads positively on GLD (inflation/safe-haven risk), and
$\text{PC}_3$ loads positively on IEF (rates/duration).
For each asset $i$ in the ETF universe, we estimate:
\begin{itemize}
\item \emph{linear betas} $\beta^{(i)}_1, \beta^{(i)}_2, \beta^{(i)}_3$ via
full-sample ordinary least squares on the three PCA factors;
\item \emph{nonlinear betas} via a polynomial expansion that includes squares
and cross-terms of the three factors, with strict per-asset caps to
prevent numerical blow-ups.
\end{itemize}
LLM macro shocks are given in percentage points as
$\Delta \mathbf{M}_s = (\Delta g_s, \Delta \pi_s, \Delta r_s)$ for
GDP growth, inflation, and the interest rate.
To connect these to the PCA factors \cite{oura2012macrofinancial}, we define the non-negative factor shock
vector in Equation~\ref{eq:factor-shock}, so that deeper recessions load more
heavily on $\text{PC}_1$, higher inflation on $\text{PC}_2$, and larger rate
hikes on $\text{PC}_3$.
\begin{equation}
\begin{aligned}
\Delta \mathbf{F}_s
&= \bigl(f^{(1)}_s, f^{(2)}_s, f^{(3)}_s\bigr) \\
&= \bigl(
\max\{0,-\Delta g_s/100\},
\max\{0,\Delta \pi_s/100\},
\max\{0,\Delta r_s/100\}
\bigr).
\end{aligned}
\label{eq:factor-shock}
\end{equation}
\paragraph{Regime-specific covariance.}
From historical ETF returns we estimate two covariance matrices:
$\Sigma_{\text{calm}}$ using a calm sample (2012--2019) and
$\Sigma_{\text{crisis}}$ using a combined GFC and COVID sample.
For each scenario $s$, the regime severity index $\lambda_s \in [0,1]$
constructed in the plausibility step defines a scenario-specific covariance:
\begin{equation}
\Sigma_{\text{scen},s}
= (1 - \lambda_s)\,\Sigma_{\text{calm}} + \lambda_s\,\Sigma_{\text{crisis}}.
\label{eq:cov-mix}
\end{equation}
Higher $\lambda_s$ values thus induce more crisis-like volatilities and
correlations while preserving a purely linear, interpretable mixture between
the two regimes \cite{gray1996modeling, HamiltonLin1996Volatility}.
\paragraph{Stress channels and simulation.}
We decompose stress transmission into three channels:
\begin{enumerate}
\item \textbf{Pure volatility channel.}
The covariance matrix $\Sigma_{\text{scen},s}$ is scaled as a deterministic
function of the inflation shock (larger inflation $\Rightarrow$ higher
volatility), while daily mean returns are kept at zero. This isolates the
impact of volatility-only amplification on VaR/CVaR.
\item \textbf{Linear PCA factor channel.}
Macroeconomic shocks are mapped to PCA factor shocks $\Delta \mathbf{F}_s$,
which are normalised by factor standard deviations and passed through the
linear betas to obtain a shocked mean return vector
$\mu_{\text{lin},s} = \mu_{\text{base}} + B \Delta \mathbf{F}_s$.
The resulting drift is applied with a geometric decay over the 63-day horizon
to reflect shock reversion.
Crucially, this channel is independent of text, retrieval, news, and $\lambda$:
it depends only on the numeric macro shocks and the pre-estimated PCA mapping.
\item \textbf{Nonlinear factor channel.}
The polynomial betas are applied to the same factor shocks to produce a
nonlinear drift adjustment \cite{Kutateladze2022KernelTrick, almeida2023nonlinear}, which is then multiplied by an amplification term
of the form
$\text{amp}_s = 1 + a_\lambda \lambda_s + a_{\text{rag}}\mathbf{1}_{\text{RAG}}
+ a_{\text{news}}\mathbf{1}_{\text{NEWS}} + \dots$,
with small fixed coefficients.
Strict per-asset caps on the resulting drifts ensure numerical stability.
This is the \emph{only} channel through which text, retrieval, and news
affect portfolio returns.
\end{enumerate}
Daily returns are simulated using a Cholesky decomposition of
$\Sigma_{\text{scen},s}$ (with jitter if needed), geometric drift decay, and
per-day clipping of extreme returns \cite{jorion1997value, glasserman2004monte, Best2000VaR}. For the volatility channel we set the mean
to zero and use the scaled $\Sigma_{\text{scen},s}$; for the linear and nonlinear
channels we use $\Sigma_{\text{scen},s}$ together with $\mu_{\text{lin},s}$ and
$\mu_{\text{lin},s} + \mu_{\text{nonlin},s}$, respectively.
In each case we simulate 20,000 paths of daily returns over a 63-day horizon.
\paragraph{Tail metrics.}
For each simulated path we compute portfolio-level losses and derive:
\begin{itemize}
\item 5\% Value-at-Risk (VaR\(_{0.95}\))~\cite{jorion1997value},
\item Conditional VaR (CVaR)~\cite{rockafellar2000optimization},
\item Maximum drawdown (MDD)~\cite{magdon2004maximum}.
\end{itemize}
These metrics are computed separately for the volatility-only, linear, and
nonlinear channels.
Severity is expressed as the ratio relative to the historical bootstrap or
econometric baseline (i.e., VaR/CVaR \emph{multiples}).
These multiples underpin the risk tables and figures reported in
Section~\ref{sec:results}, and allow us to compare LLM-induced stress against
both deterministic macro benchmarks and classical volatility models
(EWMA and GARCH-t).
\section{Experimental Setup}
\label{sec:experimental}
This section describes the experimental design used to evaluate the LLM-based
stress–testing pipeline.
Macroeconomic stress scenarios are generated for the G7 under the configuration
grid defined by the ablations in Section~\ref{sec:methodology}, filtered via
plausibility checks and regime tagging, and then mapped into portfolio tail–risk
metrics through the three stress channels described earlier.
The resulting VaR and CVaR multiples (volatility, linear, and nonlinear) are
benchmarked against historical and econometric baselines and later summarised in
the tables and figures reported in Section~\ref{sec:results}.
\subsection{Language Models and Runs}
We evaluate two instruction-tuned large language models:
\begin{itemize}
\item \textbf{GPT-5-mini} (proprietary, OpenAI), used as the main workhorse model.
\item \textbf{Llama-3.1-8B-Instruct} (open source, Hugging Face), used for cross--model comparisons.
\end{itemize}
Completions are required to emit a single JSON object; we parse the first valid object with a tolerant extractor \cite{Wu2024StructuredEntity} and discard malformed outputs (rare, $<2\%$).
\paragraph{Model and hardware details.}
GPT–5–mini is accessed through the OpenAI API and
Llama–3.1–8B–Instruct via the Hugging Face
Inference API. We do not apply any custom quantisation, fine–tuning,
or weight modifications: both models are used exactly as provided by their
respective hosted endpoints. We rely on the providers’ default inference
settings (including their default sampling parameters and seed behaviour) and
do not enforce strict greedy decoding. All prompts, including retrieved
context, are constructed to remain well within the provider–exposed context
windows, and we therefore do not impose any additional truncation rules
beyond the usual safety checks for malformed JSON.
For retrieval\textendash augmented generation (RAG), we deliberately fix the
retrieval depth at top–$k=3$ documents. This choice reflects a balance between
contextual richness, latency, and prompt readability, rather than any hard
capacity constraint imposed by the models or infrastructure. Inference runs on
provider\textendash hosted CPU endpoints (no GPU acceleration), but the pipeline
itself is hardware\textendash agnostic, and all experiments operate comfortably
within the resource limits of standard hosted inference services.
We run three complementary experiments:
\begin{enumerate}
\item A \emph{deterministic GPT-5-mini run} (\texttt{run\_det}), which forms the core dataset for the macro, risk, stability, and fairness results in Section~\ref{sec:results}.
\item A \emph{non-deterministic GPT-5-mini run} (\texttt{run\_nondet}) with the same configuration grid but allowing internal stochasticity (e.g., sampling in upstream retrieval or simulation loops).
\item A \emph{non-deterministic Llama-3.1-8B-Instruct run} (\texttt{run\_llama}) used for cross--model severity and risk comparisons, reported alongside the deterministic baseline in Section~\ref{sec:results}.
\end{enumerate}
Across the three runs we obtain 627 (deterministic GPT-5-mini), 617
(non-deterministic GPT-5-mini), and 307 (Llama-3.1-8B-Instruct) accepted
scenarios, respectively, after plausibility filtering via the two-layer audit
described in Section~\ref{sec:methodology}.
These accepted scenarios form the inputs to the volatility, linear, and
nonlinear stress channels used in the risk evaluation.
\paragraph{Run attribution.}
Unless explicitly stated otherwise, all tables and figures in the main text are based on the deterministic GPT-5-mini run (\texttt{run\_det}). Results that use the non-deterministic GPT-5-mini run (\texttt{run\_nondet}) or the Llama-3.1-8B-Instruct run (\texttt{run\_llama}) are explicitly identified in the surrounding text or figure/table captions.
\subsection{Configuration Grid and Scenario Sampling}
The main deterministic run uses a full factorial grid over:
\begin{itemize}
\item \textbf{Countries:} G7 (Canada, France, Germany, Italy, Japan, United Kingdom, United States).
\item \textbf{Retrieval (RAG):} on vs.\ off.
\item \textbf{News retrieval:} on vs.\ off.
\item \textbf{Prompt variant:} 30 manually designed macro narratives per country.
\end{itemize}
For each (country, RAG, news) configuration we generate one scenario per prompt variant, yielding
\[
7\ \text{countries} \times 4\ \text{configs} \times 30\ \text{prompts} = 840
\]
intended scenarios per model.
After filtering out malformed outputs and scenarios failing the hard or soft
plausibility gates, the deterministic GPT-5-mini run yields between 83 and 95
accepted scenarios per country (Table~\ref{tab:macro-summary}, last column), for
a total of 627 accepted scenarios.
Scenario stability is quantified ex post using two dispersion metrics derived from these samples:
\begin{itemize}
\item \textbf{Intra-prompt dispersion} (Table~\ref{tab:prompt-dispersion}): for each prompt variant and configuration, we pool the accepted scenarios for that prompt across runs and compute the average pairwise distance between macro-shock vectors
(gdp\_growth, inflation, interest\_rate).
\item \textbf{Intra-configuration dispersion} (Table~\ref{tab:stability-config}): for each (country, RAG, news) cell in the deterministic run, we compute the average pairwise distance across all accepted scenarios in that cell, with bootstrap confidence intervals.
\end{itemize}
\paragraph{Dispersion metric.}
For a given cell (e.g., a fixed country, model, RAG, and news configuration)
with $S$ accepted scenarios, let
$\mathbf{x}_s = (\Delta g_s, \Delta \pi_s, \Delta r_s)$ denote the vector of
GDP, inflation, and interest-rate shocks (in percentage points) for scenario $s$.
We measure stability using the dispersion metric in
Equation~\ref{eq:dispersion}, a standard approach based on mean
pairwise Euclidean distance,
commonly used to assess diversity and robustness in generative models
and scenario generators \cite{FreieslebenGrote2023Robustness}.
\begin{equation}
D = \frac{2}{S(S-1)} \sum_{1 \le i < j \le S}
\left\lVert \mathbf{x}_i - \mathbf{x}_j \right\rVert_2.
\label{eq:dispersion}
\end{equation}
No additional scaling or standardisation is applied; all shocks are measured in
percentage points for comparability with
Tables~\ref{tab:macro-summary} and~\ref{tab:severity-model}.
This metric is used both at the prompt level (Table~\ref{tab:prompt-dispersion})
and at the configuration level (Table~\ref{tab:stability-config}).
One corrupted configuration and one corrupted prompt-level row with extremely large dispersion were removed via simple QC filters (threshold 20 in the shock space), as reported in the table logs.
\subsection{Portfolio and Econometric Baselines}
All scenarios are mapped to two stylised ETF portfolios:
\begin{itemize}
\item \textbf{Portfolio A} (headline portfolio):
\begin{itemize}
\item \textbf{SPY} — U.S.\ equities (S\&P~500 proxy) — 60\% weight,
\item \textbf{IEF} — intermediate-term U.S.\ Treasuries — 30\% weight,
\item \textbf{GLD} — gold bullion — 10\% weight.
\end{itemize}
\item \textbf{Portfolio B} (sector-tilted robustness portfolio): an equal-weighted portfolio across sector ETFs with full history (XLE, XLF, XLK, XLY, XLI, XLU, XLV, XLP, XLB, XLRE).
\end{itemize}
Weights are re-normalised daily over a 63-day horizon (one quarter) to maintain
constant-weight exposure.
Portfolio~A is the main focus of the paper; Portfolio~B is used to test
cross--portfolio robustness in the fairness diagnostics
(Table~\ref{tab:fairness}).
To provide non-LLM risk baselines for Portfolio~A, we estimate 63-day VaR and
CVaR under three standard methods (Table~\ref{tab:baseline-var}; see also
Figure~\ref{fig:baselines} in Section~\ref{sec:results}).
All three are constructed from overlapping 63-day windows of historical returns:
\begin{itemize}
\item \textbf{Historical Baseline (Bootstrap)} on daily returns over 2000--2025,
sampling overlapping 63-day blocks with replacement \cite{efron1992bootstrap, politis1994stationary}.
\item \textbf{EWMA} with decay factor $\lambda = 0.94$ under a Normal assumption, rescaled to the 63-day horizon.
\item \textbf{GARCH(1,1)–t} fitted to daily returns, with 63-day losses obtained from simulated paths.
\end{itemize}
The resulting 63-day loss estimates (decimal units) are approximately:
\begin{table}[t]
\centering
\caption{Portfolio A: 63-day VaR and CVaR under historical and econometric baselines.}
\label{tab:baseline-var}
\begin{tabular}{lcc}
\toprule
Method &
\makecell{VaR$_{0.95}$\\(decimal loss)} &
\makecell{CVaR$_{0.95}$\\(decimal loss)} \\
\midrule
Historical Baseline (Bootstrap) & 0.0491 & 0.0932 \\
EWMA ($\lambda=0.94$, Normal) & 0.0725 & 0.0909 \\
GARCH(1,1)–t (Simulated) & 0.0856 & 0.1202 \\
\bottomrule
\end{tabular}
\end{table}
By default we treat the 2000--2025 historical bootstrap as the reference
baseline for LLM multiples.
The more flexible EWMA and GARCH models serve as a robustness ladder indicating
how far LLM-induced stress lies above simple historical benchmarks.
Section~\ref{sec:crisis-envelopes} additionally reports crisis envelopes for
Portfolio~A by comparing GFC and COVID windows against both this unconditional
baseline and a calm-period (2012--2019) bootstrap; these envelopes are used only
for historical episode benchmarking and do not replace the main baseline in
Table~\ref{tab:baseline-var} for LLM-generated scenario multiples.
\subsection{Shock Translation and Tail-Risk Metrics}
For each accepted scenario $s$, the LLM produces macro shocks in percentage points for real GDP growth, inflation, and the interest rate.
These shocks are mapped into portfolio tail risk via the three-channel factor
model described in Section~\ref{sec:methodology}.
In brief, we estimate linear and nonlinear PCA factor betas for each asset,
construct a scenario-specific covariance matrix
$\Sigma_{\text{scen},s} = (1-\lambda_s)\Sigma_{\text{calm}} +
\lambda_s\Sigma_{\text{crisis}}$, and simulate 63-day paths under:
\begin{itemize}
\item a \emph{pure volatility} channel (zero drift, scaled covariance),
\item a \emph{linear} channel (drift from the linear PCA mapping only),
\item a \emph{nonlinear} channel (drift from linear + polynomial terms,
amplified by $\lambda_s$ and the RAG/news flags).
\end{itemize}
Let $\text{VaR}^{\text{base}}$ and $\text{CVaR}^{\text{base}}$ denote the
63-day VaR and CVaR of Portfolio~A under the historical bootstrap baseline, and
let $\text{VaR}^{\text{(ch)}}_s$ and $\text{CVaR}^{\text{(ch)}}_s$ be the
corresponding quantities under scenario $s$ in a given channel
$\text{ch} \in \{\text{vol}, \text{lin}, \text{nonlin}\}$.
We compute VaR and CVaR multiples using the definitions in
Equation~\ref{eq:var-multiple}, which express scenario losses relative to the
historical baseline.
\begin{equation}
\begin{aligned}
\text{VaR multiple}^{\text{(ch)}}_s &=
\frac{\text{VaR}^{\text{(ch)}}_s}{\text{VaR}^{\text{base}}}, \\
\Delta\text{VaR}\%^{\text{(ch)}}_s &=
100 \times
\frac{\text{VaR}^{\text{(ch)}}_s - \text{VaR}^{\text{base}}}{|\text{VaR}^{\text{base}}|}.
\end{aligned}
\label{eq:var-multiple}
\end{equation}
An analogous definition is used for CVaR multiples.
Unless otherwise stated, summary tables in the main text report the
\emph{linear-channel} multiples, denoted
$\text{VaR multiple}^{\text{(lin)}}_s$ and
$\text{CVaR multiple}^{\text{(lin)}}_s$, while volatility- and nonlinear-channel
results are provided alongside them in the risk tables
(e.g., Table~\ref{tab:risk-crossrun}) and the appendix.
Table~\ref{tab:risk-crossrun} summarises the mean and standard deviation of
VaR and CVaR multiples by model, RAG, and news configuration for each channel,
while Figures~\ref{fig:cvar-by-country}, \ref{fig:inflation-vs-cvar},
\ref{fig:news-effect}, and~\ref{fig:heatmaps} visualise their distribution
across countries, channels, and RAG/news settings.
Bootstrap confidence intervals for linear-channel multiples by
(model, RAG, news) are reported in Table~\ref{tab:boot-cis}.
\subsection{Configuration Grid for Risk Evaluation}
For risk evaluation we work with the same $(\text{country}, \text{RAG}, \text{news})$ grid as for generation, yielding four configurations per country in the deterministic GPT-5-mini run.
For each configuration we compute the distribution of VaR and CVaR multiples
for Portfolios~A and~B and for each of the three stress channels, as well as
stability and fairness diagnostics:
\begin{itemize}
\item \textbf{Country–level macro shock summary} (Table~\ref{tab:macro-summary} and Figure~\ref{fig:macro-dists}),
\item \textbf{Portfolio A CVaR multiples by country and channel} (Figure~\ref{fig:cvar-by-country}),
\item \textbf{News-on vs.\ news-off effects} (Figure~\ref{fig:news-effect}),
\item \textbf{Cross–country gaps in average multiples and fairness metrics} (Table~\ref{tab:fairness}).
\end{itemize}
Cross–run comparisons of scenario severity and risk (GPT-5-mini vs.\ Llama-3.1-8B-Instruct) are reported in Table~\ref{tab:severity-model} and Figure~\ref{fig:model-comparison}.
\subsection{Statistical and Fairness Diagnostics}
To understand which design choices drive variation in tail risk, we run an ANOVA \cite{fisher1970statistical} on the linear-channel VaR and CVaR multiples using the following categorical factors:
\begin{itemize}
\item \texttt{portfolio\_id} (A vs.\ B),
\item \texttt{country} (seven G7 economies),
\item \texttt{prompt\_variant} (30 macro narratives),
\item \texttt{rag} (on/off),
\item \texttt{use\_news} (on/off).
\end{itemize}
Table~\ref{tab:anova-decomp} reports, for each effect, the corresponding $p$-value and partial $\eta^2$ \cite{cohen2013statistical} for both VaR and CVaR multiples.
In parallel, we compute a set of fairness and robustness diagnostics:
\begin{itemize}
\item \textbf{Coverage and outliers} (Table~\ref{tab:fairness}): rows with
full coverage, label flips under small perturbations, and outliers
detected by both standard and robust (MAD-based) $z$-scores \cite{rabonato2025systematic}.
\item \textbf{Country gaps} (Table~\ref{tab:fairness}): max--min differences
in mean VaR and CVaR multiples across countries for each portfolio.
\item \textbf{Linear vs.\ nonlinear channels}: the ``linear'' gaps use the
VaR/CVaR multiples implied by the purely linear PCA mapping
(Section~\ref{sec:methodology}), while the ``nonlinear'' gaps use
multiples from the full nonlinear-factor channel with polynomial betas
and $\lambda$-driven amplification. Nonlinear gaps are numerically
small relative to linear gaps, and we therefore focus on the linear
channel in the main text.
\item \textbf{Flip tests} (news on vs.\ off) and outlier flags, used later in
the results section to check that RAG/news toggles do not introduce
unstable or systematically skewed shifts in tail risk.
\end{itemize}
These diagnostics ensure that our conclusions are not driven by a small number
of unstable or unfair configurations, and they quantify how much of the variance
in tail risk is attributable to portfolio composition, scenario design,
country, and retrieval settings.
\subsection{Compute and Reproducibility}
All experiments are run on commodity CPU instances and T4 GPU, ensuring a hardware agnostic pipeline. We log per–call latency and token usage for each model,
as well as a lightweight run manifest that hashes all critical artifacts:
\begin{itemize}
\item WEO baselines and scenario files (CSV/JSON),
\item cached market data (ETF prices) and derived PCA factors,
\item FAISS indices and MiniLM weights,
\item headline CSV snapshots for news-enabled runs,
\item prompts and parsed JSON responses,
\item risk tables and figure/table artifacts.
\end{itemize}
For each run we write a compact manifest
(\texttt{run\_artifacts\_index.json}) containing the run identifier,
workspace tag, model configuration, news/RAG settings, and SHA-256 hashes plus
row counts for all key files. This ensures that any reported figure or table
can be traced back to a fixed set of documents, embeddings, scenarios, and
risk computations.
For news-enabled configurations, headline snapshots are fetched once, written
to CSV with timestamps, and reused throughout the experiment. The manifest
records their paths and hashes, ensuring that the exact news context used at
inference time can be replayed without re-querying external APIs.
We emphasise that the pipeline is \emph{snapshot-replayable}: given the frozen
artifacts (IMF baselines, cached ETF prices, headline CSVs, MiniLM weights,
FAISS index, prompts, and global random seeds), all macro scenarios and
portfolio risk metrics can be regenerated up to floating-point and Monte Carlo
noise. Deterministic decoding stabilizes model outputs conditional on the
retrieved context, and the deterministic historical baselines and PCA factors
eliminate external data drift, but strict hardware-agnostic bit-level
determinism is not claimed.
\section{Results}
\label{sec:results}
The main quantitative results are summarised in the macro, severity, tail-risk, stability, and fairness tables and figures throughout this section.
Additional robustness checks and expanded statistics are deferred to the
Appendix.
\subsection{Macroeconomic Shock Distributions}
LLM-generated macro shocks exhibit clear country structure and appropriate
stress polarity.
Figure~\ref{fig:macro-dists} shows the distributions of GDP, inflation, and interest rate shocks across all
scenarios.
\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{fig01_macro_shock_distributions.png}
\caption{Violin plots of GDP, inflation, and interest rate shocks
(percentage points) for all accepted G7 scenarios in the deterministic
GPT-5-mini run ($N=627$).
Each panel shows the full distribution by country; medians and
interquartile ranges correspond to the summary statistics in
Table~\ref{tab:macro-summary}.}
\label{fig:macro-dists}
\end{figure*}
Table~\ref{tab:macro-summary} provides the aggregated summary by country.
GDP shocks are uniformly negative---typically around $-1.3$ to $-1.5$\,pp---while
inflation shocks are moderately positive (roughly $+3$\,pp).
Interest rate shocks vary significantly across countries:
Japan exhibits small responses (mean $\approx1.3$\,pp),
whereas the U.S.\ and U.K.\ exhibit much larger increases ($\approx5$\,pp).
Scenario counts per country (roughly 83--95 per G7 member) confirm balanced coverage after plausibility filtering.
\begin{table*}[t]
\centering
\caption{LLM-generated macro shocks by country (GDP, inflation, interest rate; deterministic GPT-5-mini run).}
\label{tab:macro-summary}
\begin{tabular}{lccccccccccccc}
\toprule
& \multicolumn{4}{c}{GDP shock} & \multicolumn{4}{c}{Inflation shock} & \multicolumn{4}{c}{interest rate shock} & \\
\cmidrule(lr){2-5}\cmidrule(lr){6-9}\cmidrule(lr){10-13}
Country & Mean & Std & Min & Max & Mean & Std & Min & Max & Mean & Std & Min & Max & $N$ \\
\midrule
Canada & -1.42 & 0.77 & -4.20 & -0.80 & 3.31 & 1.66 & 0.0 & 6.5 & 4.33 & 1.52 & 0.00 & 6.75 & 95 \\
France & -1.37 & 0.71 & -3.80 & -0.80 & 2.83 & 1.87 & -0.3 & 6.8 & 3.52 & 1.54 & 0.25 & 6.25 & 86 \\
Germany & -1.53 & 0.76 & -4.50 & -0.80 & 3.45 & 1.69 & 0.6 & 6.8 & 3.51 & 1.42 & 0.00 & 5.75 & 95 \\
Italy & -1.52 & 0.77 & -3.80 & -0.80 & 3.12 & 1.90 & -0.6 & 6.8 & 3.82 & 1.50 & 0.50 & 7.50 & 94 \\
Japan & -1.35 & 0.66 & -3.80 & -0.80 & 2.67 & 1.85 & -0.8 & 6.4 & 1.33 & 1.34 & -0.10 & 6.25 & 83 \\
United Kingdom & -1.38 & 0.72 & -4.20 & -0.80 & 3.73 & 1.79 & 0.6 & 7.8 & 5.02 & 1.47 & 0.25 & 6.75 & 91 \\
United States & -1.42 & 0.67 & -4.20 & -0.80 & 3.36 & 1.82 & -0.7 & 6.5 & 4.64 & 1.92 & -0.75 & 7.25 & 83 \\
\bottomrule
\end{tabular}
\end{table*}
\subsection{Scenario Severity by Model}
Table~\ref{tab:severity-model} summarizes unconditional macro severity
by model across all countries and configurations.
Llama-3.1-8B-Instruct delivers slightly deeper GDP contractions on average
($-1.67$\,pp vs.\ $-1.44$\,pp for GPT-5-mini), while GPT-5-mini produces
higher interest-rate shocks ($3.76$\,pp vs.\ $2.64$\,pp).
Mean absolute macro shock magnitude is somewhat larger for GPT-5-mini
($2.81$ vs.\ $2.52$), reflecting its stronger rate moves, whereas Llama leans
more into real-side pain (GDP).
Figure~\ref{fig:model-comparison} visualizes these differences and
shows that despite these contrasts in macro severity, the resulting
\emph{linear-channel} portfolio CVaR multiples are of similar order of magnitude,
with Llama producing slightly fatter tails.
\begin{table*}[t]
\centering
\caption{Scenario severity by model (unconditional macro shock magnitudes across all countries and configurations).}
\label{tab:severity-model}
\begin{tabular}{lcccc}
\toprule
Model & Mean GDP shock & Mean inflation shock & Mean interest rate shock & Mean $|\text{macro shock}|$ \\
\midrule
GPT-5-mini & -1.44 & 3.19 & 3.76 & 2.81 \\
Llama-3.1-8B-Instruct & -1.67 & 3.23 & 2.64 & 2.52 \\
\bottomrule
\end{tabular}
\end{table*}
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{fig06_model_comparison_severity_and_cvar.png}
\caption{Comparison of average macro shock severity (left; mean absolute
GDP, inflation, and interest rate shocks) and linear CVaR multiples (right)
by model (GPT-5-mini vs.\ Llama-3.1-8B-Instruct), pooling over all
countries and configurations (see Tables~\ref{tab:severity-model}
and~\ref{tab:risk-crossrun}).}
\label{fig:model-comparison}
\end{figure}
\subsection{Portfolio Tail-Risk Multiples}
Table~\ref{tab:risk-crossrun} reports mean VaR and CVaR multiples across
models and retrieval configurations for the \emph{linear} channel, together with
their standard deviations.
Bootstrap confidence intervals for these linear-channel multiples appear in
Table~\ref{tab:boot-cis} (Appendix~A.1).
For GPT-5-mini, linear VaR multiples range from roughly $1.46$ to $1.48\times$
and linear CVaR multiples are tightly clustered around $1.13\times$ across
RAG/news settings.
Llama-3.1-8B-Instruct produces slightly lower linear VaR multiples
($\approx1.41$--$1.42\times$) but higher linear CVaR multiples
($\approx1.22$--$1.23\times$).
The bootstrap intervals are uniformly tight—typically within $\pm 0.01$ of the mean—highlighting the stability of the tail-risk
estimates across scenario realisations.
Beyond the linear channel, the volatility channel implies VaR multiples of
about $3.6$--$3.8\times$ and CVaR multiples of about $2.7$--$3.0\times$
relative to the historical bootstrap baseline, while the nonlinear channel adds
a modest incremental amplification: nonlinear CVaR multiples are roughly
$1.07$--$1.08\times$ for GPT-5-mini and $1.15$--$1.17\times$ for Llama,
i.e., an additional 7--17\% above the linear channel.
\begin{table*}[t]
\centering
\hspace*{-0.5cm}
\begin{minipage}{1.1\textwidth}
\centering
\caption{Portfolio tail risk by model, retrieval (RAG), and news configuration (cross-run averages across all countries and portfolios). Reported values are VaR/CVaR multiples relative to the historical bootstrap baseline for each stress channel.}
\label{tab:risk-crossrun}
\begin{tabular}{lcc
cc cc
cc cc
cc cc
c}
\toprule
Model & RAG & News &
\multicolumn{2}{c}{\makecell{VaR multiple\\(vol.)}} &
\multicolumn{2}{c}{\makecell{CVaR multiple\\(vol.)}} &
\multicolumn{2}{c}{\makecell{VaR multiple\\(linear)}} &
\multicolumn{2}{c}{\makecell{CVaR multiple\\(linear)}} &
\multicolumn{2}{c}{\makecell{VaR multiple\\(nonlin.)}} &
\multicolumn{2}{c}{\makecell{CVaR multiple\\(nonlin.)}} &
$N$ \\
\cmidrule(lr){4-5}\cmidrule(lr){6-7}
\cmidrule(lr){8-9}\cmidrule(lr){10-11}
\cmidrule(lr){12-13}\cmidrule(lr){14-15}
& & &
Mean & Std & Mean & Std &
Mean & Std & Mean & Std &
Mean & Std & Mean & Std & \\
\midrule
GPT-5-mini & Off & Off & 3.79 & 0.46 & 2.74 & 0.39 & 1.46 & 0.06 & 1.13 & 0.10 & 1.38 & 0.05 & 1.08 & 0.09 & 288 \\
GPT-5-mini & Off & On & 3.76 & 0.45 & 2.72 & 0.38 & 1.47 & 0.06 & 1.13 & 0.10 & 1.37 & 0.05 & 1.08 & 0.09 & 299 \\
GPT-5-mini & On & Off & 3.76 & 0.48 & 2.72 & 0.40 & 1.46 & 0.06 & 1.13 & 0.10 & 1.38 & 0.05 & 1.08 & 0.09 & 336 \\
GPT-5-mini & On & On & 3.74 & 0.48 & 2.70 & 0.40 & 1.48 & 0.07 & 1.13 & 0.10 & 1.36 & 0.05 & 1.07 & 0.09 & 321 \\
Llama-3.1-8B-Instruct & Off & Off & 3.56 & 0.38 & 2.88 & 0.30 & 1.42 & 0.04 & 1.23 & 0.02 & 1.34 & 0.01 & 1.17 & 0.00 & 77 \\
Llama-3.1-8B-Instruct & Off & On & 3.64 & 0.37 & 2.94 & 0.29 & 1.42 & 0.04 & 1.23 & 0.02 & 1.33 & 0.01 & 1.17 & 0.01 & 74 \\
Llama-3.1-8B-Instruct & On & Off & 3.70 & 0.41 & 2.99 & 0.31 & 1.41 & 0.04 & 1.22 & 0.02 & 1.33 & 0.01 & 1.17 & 0.00 & 80 \\
Llama-3.1-8B-Instruct & On & On & 3.68 & 0.41 & 2.98 & 0.32 & 1.41 & 0.04 & 1.22 & 0.02 & 1.31 & 0.01 & 1.15 & 0.01 & 76 \\
\bottomrule
\end{tabular}
\end{minipage}
\end{table*}
Figure~\ref{fig:cvar-by-country} further breaks down \emph{linear} CVaR multiples
by country for Portfolio A.
The 1.10--1.20 range is common across the G7, with modestly higher dispersion for
Japan, Italy, and France.
This confirms that despite differences in macro narratives, the linear tail-risk
amplification is relatively uniform across geographies.
\begin{figure*}[t]
\centering
\includegraphics[width=\linewidth]{fig02_cvar_multiples_by_country_all_channels.png}
\caption{Boxplots of linear CVaR multiples for Portfolio~A by country,
pooling over all model/RAG/news configurations in the deterministic run.
Values are expressed as multiples relative to the historical-bootstrap
baseline; see Table~\ref{tab:risk-crossrun}.}
\label{fig:cvar-by-country}
\end{figure*}
\subsection{Relationship Between Macro Shocks and Tail Risk}
Figure~\ref{fig:inflation-vs-cvar} plots inflation shocks against
scenario-induced CVaR multiples for Portfolio~A across all three
risk-translation channels: volatility, linear, and nonlinear.
Across all panels, the relationship remains weakly negative or near-flat:
larger inflation shocks do not mechanically produce larger tail risk,
even though inflation helps govern volatility scaling in the pure
volatility channel.
Country-specific clustering is visible in each panel (e.g., Japan exhibits
relatively high inflation shocks but only moderate CVaR multiples, whereas
the United States and United Kingdom show somewhat wider upper tails).
The volatility channel exhibits the largest dispersion, the linear channel
the tightest, and the nonlinear channel remains close to unity with thin
tails.
Overall, the figure demonstrates that tail risk depends on the \emph{joint}
interaction of macro shocks, regime severity~$\lambda_s$, channel choice,
and portfolio exposures—not on inflation alone.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{fig03_inflation_vs_cvar_linear.png}
\caption{Scenario-induced CVaR multiples for Portfolio~A plotted against
inflation shocks (percentage points) across the three translation channels:
volatility (left), linear (centre), and nonlinear (right). Each point is a
scenario; colours (in the online version) indicate country. The weak
relationship across all three panels highlights that tail risk emerges
from the joint macro shock vector, regime mixing, and portfolio composition
rather than inflation in isolation.}
\label{fig:inflation-vs-cvar}
\end{figure}
\subsection{Effect of News Retrieval}
Figure~\ref{fig:news-effect} compares linear CVaR multiples for GPT-5-mini
with news retrieval on versus off (RAG enabled).
Median effects are small, but the news-enabled distribution has a slightly
wider upper tail—consistent with the ANOVA finding that news has a small but
statistically significant effect size ($\eta^2\approx0.014$;
Table~\ref{tab:anova-decomp}).
This pattern is mirrored in the nonlinear channel, where the amplification
term includes a small positive coefficient on the news flag.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{fig04_news_effect_on_cvar_linear.png}
\caption{Distribution of linear CVaR multiples for Portfolio~A under
GPT-5-mini with RAG enabled, comparing scenarios with and without news
retrieval.
The news-enabled distribution has a slightly wider upper tail, consistent
with the small but statistically significant news effect in
Table~\ref{tab:anova-decomp}.}
\label{fig:news-effect}
\end{figure}
\subsection{Baselines for Historical and Econometric Risk}
Figure~\ref{fig:baselines} and Table~\ref{tab:baseline-var} summarize the
historical bootstrap, EWMA, and GARCH-t VaR/CVaR benchmarks used throughout the
paper.
Risk estimates increase with model flexibility, with GARCH-t producing the most
conservative baseline.
\begin{figure*}[t]
\centering
\includegraphics[width=\linewidth]{fig05_var_cvar_benchmarks.png}
\caption{63-day VaR and CVaR (decimal losses) for Portfolio~A under three
baseline methods: historical bootstrap, EWMA ($\lambda=0.94$), and
GARCH(1,1)--t.
Numerical values are reported in Table~\ref{tab:baseline-var}.}
\label{fig:baselines}
\end{figure*}
\subsection{Historical Crisis Envelopes and Calm-Period Baselines}
\label{sec:crisis-envelopes}
To benchmark LLM-generated scenarios against realized crisis dynamics, we compute
63-day historical crisis envelopes for Portfolio~A using the GFC (2008--2009)
and COVID-19 (2020) windows. All estimates use a fixed 63-day horizon to remain
consistent with the simulation engine and econometric baselines in
Table~\ref{tab:baseline-var}.
\paragraph{Unconditional baseline (2000--2025).}
Using the unconditional 2000–2025 baseline in
Equation~\ref{eq:baseline-uncond}, we obtain 63-day VaR, CVaR, and MDD values
that anchor historical crisis envelopes.
\begin{equation}
\begin{aligned}
\text{VaR}_{0.95} &= 4.91\%, \\
\text{CVaR}_{0.95} &= 9.32\%, \\
\text{MDD} &= -4.34\%.
\end{aligned}
\label{eq:baseline-uncond}
\end{equation}
The GFC and COVID-19 crisis envelopes derived from this distribution are shown
in Equation~\ref{eq:crisis-uncond}, providing empirical upper and lower
comparators for LLM-generated stress.
\begin{equation}
\begin{aligned}
\text{GFC:}\;& \text{VaR}\times = 3.46,\quad \text{CVaR}\times = 2.07, \\
\text{COVID:}\;& \text{VaR}\times = 0.99,\quad \text{CVaR}\times = 0.52.
\end{aligned}
\label{eq:crisis-uncond}
\end{equation}
Because the unconditional baseline already contains the 2008 and 2020 extremes,
it is intrinsically fat-tailed. Thus the GFC appears very severe relative to it,
while COVID-19 appears moderate in comparison---a mechanically correct result
given that 2008 dominates the full-sample tail.
\paragraph{Calm-period baseline (2012--2019).}
To isolate structural crisis severity from unconditional tail thickness, we
construct a calm-period baseline excluding both major crises. We also compute a calm-period baseline, given in
Equation~\ref{eq:baseline-calm}, which excludes major crisis windows and
provides a lower-volatility reference for scenario comparison.
\begin{equation}
\begin{aligned}
\text{VaR}_{0.95} &= 2.83\%, \\
\text{CVaR}_{0.95} &= 4.34\%, \\
\text{MDD} &= -3.00\%.
\end{aligned}
\label{eq:baseline-calm}
\end{equation}
Relative to the calm-period baseline, the crisis envelopes in
Equation~\ref{eq:crisis-calm} indicate substantially amplified tail risk during
the GFC and moderately elevated tail risk during COVID-19.
\begin{equation}
\begin{aligned}
\text{GFC:}\;& \text{VaR}\times = 6.00,\quad \text{CVaR}\times = 4.45, \\
\text{COVID:}\;& \text{VaR}\times = 1.71,\quad \text{CVaR}\times = 1.12.
\end{aligned}
\label{eq:crisis-calm}
\end{equation}
This baseline aligns more closely with narrative expectations: the GFC produces
a 4--6$\times$ tail amplification, whereas COVID produces a 1.1--1.7$\times$
uplift. These values provide an empirical anchor for contextualising the
LLM-generated VaR/CVaR multiples reported throughout
Section~\ref{sec:results}, which for the linear channel fall mostly in the
1.1--1.5$\times$ range, comfortably below the historical crisis envelopes.
\subsection{Extreme Scenarios}
Table~\ref{tab:top10} lists the ten most severe scenarios by \emph{linear}
CVaR multiple across all runs and configurations.
The tail is dominated by (i) public-health resurgence scenarios in North America
and Europe, (ii) financial contagion and commodity-price crashes affecting
Japan, and (iii) policy-constraint scenarios in which inflation remains high
despite limited policy space.
GPT-5-mini accounts for nine of the top ten scenarios, with one
Llama-3.1-8B-Instruct policy-constraint shock also entering the top ten.
CVaR multiples in this tail lie in a narrow 1.31--1.35$\times$ band, indicating
that even the worst LLM-generated linear-channel scenarios remain far below
historical GFC-type amplifications.
\begin{table*}[t]
\centering
\caption{Ten most severe scenarios ranked by CVaR multiple (Portfolio A, linear factor channel, cross-run).}
\label{tab:top10}
\begin{tabularx}{\textwidth}{lcccccX}
\toprule
Country &
Model &
RAG &
News &
\makecell{Prompt\\variant} &
\makecell{CVaR multiple\\vs baseline} &
Scenario title \\
\midrule
Canada & GPT-5-mini & Off & On & v27\_public\_health\_resurgence
& 1.35 & Renewed Respiratory Virus Outbreak Prompts Regional Mobility Restrictions and Service Disruptions \\
United States & GPT-5-mini & On & On & v27\_public\_health\_resurgence
& 1.35 & Severe Seasonal Respiratory Outbreak Triggers Mobility Curbs and Service Disruptions \\
Japan & GPT-5-mini & On & Off & v10\_contagion
& 1.34 & Financial Contagion Scenario \\
Japan & GPT-5-mini & On & Off & v25\_commodity\_price\_crash
& 1.33 & Commodity Price Collapse Hits Japan: Metals, Energy and Agriculture Slump Drag Economy into Contraction \\
Japan & GPT-5-mini & On & Off & v20\_bigtech\_disruption
& 1.33 & AI-Accelerated Manufacturing Reallocation and Service Automation Shock (Q4--2026) \\
France & GPT-5-mini & On & On & v25\_commodity\_price\_crash
& 1.32 & Sudden Commodity Price Collapse Shocks French Economy (Metals, Energy and Agriculture) \\
France & GPT-5-mini & Off & On & v25\_commodity\_price\_crash
& 1.32 & Commodity Price Collapse --- Metals, Energy and Agricultural Prices Plunge \\
Germany & GPT-5-mini & On & Off & v27\_public\_health\_resurgence
& 1.31 & Severe Influenza-like Respiratory Outbreak Triggers Mobility Curbs and Service Disruptions \\
France & Llama-3.1-8B-Instruct & On & On & v12\_policy\_constraint
& 1.31 & High Inflation with Constrained Policy Rates \\
Japan & GPT-5-mini & On & On & v12\_policy\_constraint
& 1.31 & Q4--2026 Stress Scenario: Stubborn Inflation with Constrained Policy Rates \\
\bottomrule
\end{tabularx}
\end{table*}
\subsection{Prompt Dispersion and Stability}
Table~\ref{tab:prompt-dispersion} summarizes dispersion across the 30
prompt variants per configuration for GPT-5-mini.
Mean dispersion lies between 1.9 and 2.2 across RAG/news settings, with one
high-dispersion outlier removed by a simple QC filter (threshold 20 in shock
space, as noted in the table log).
This confirms moderate variability and underscores the importance of prompt
design, but also shows that dispersion remains bounded under prompt changes.
\begin{table*}[t]
\centering
\caption{Prompt-level dispersion of macro shocks by model, RAG, and news (average distance in shock space across repeats; deterministic GPT-5-mini run).}
\label{tab:prompt-dispersion}
\begin{tabular}{lccccccc}
\toprule
Model & RAG & News & Mean dispersion & Std dispersion & Min dispersion & Max dispersion & $N$ prompts \\
\midrule
GPT-5-mini & Off & Off & 2.217 & 0.811 & 0.579 & 3.896 & 30 \\
GPT-5-mini & Off & On & 2.137 & 0.609 & 1.193 & 3.521 & 30 \\
GPT-5-mini & On & Off & 1.895 & 0.637 & 0.338 & 3.464 & 30 \\
GPT-5-mini & On & On & 2.114 & 0.828 & 0.953 & 4.640 & 29 \\
\bottomrule
\end{tabular}
\end{table*}
Table~\ref{tab:stability-config} reports scenario stability within
(country, RAG, news) cells.
Intra-config dispersion is generally between 2.5 and 3.6, with Japan and the U.S.
showing slightly higher variability when RAG and news are enabled.
Tight bootstrap intervals indicate these estimates are statistically well-determined.
\begin{table*}[t]
\centering
\caption{Scenario stability by country and configuration (intra-configuration macro-shock dispersion with bootstrap confidence intervals; deterministic GPT-5-mini run).}
\label{tab:stability-config}
\begin{tabular}{lccccrc}
\toprule
Country & RAG & News & Intra-config dispersion & CI low & CI high & $N$ scenarios \\
\midrule
Canada & Off & Off & 3.098 & 2.955 & 3.255 & 30 \\
Canada & Off & On & 2.711 & 2.569 & 2.865 & 30 \\
Canada & On & Off & 2.732 & 2.587 & 2.878 & 30 \\
Canada & On & On & 3.008 & 2.843 & 3.201 & 30 \\
France & Off & Off & 2.799 & 2.660 & 2.943 & 30 \\
France & Off & On & 3.015 & 2.871 & 3.157 & 30 \\
France & On & Off & 2.924 & 2.787 & 3.083 & 30 \\
France & On & On & 3.081 & 2.925 & 3.231 & 30 \\
Germany & Off & Off & 2.933 & 2.790 & 3.070 & 30 \\
Germany & Off & On & 2.588 & 2.451 & 2.737 & 30 \\
Germany & On & Off & 3.310 & 3.162 & 3.464 & 30 \\
Germany & On & On & 3.045 & 2.888 & 3.220 & 30 \\
Italy & Off & Off & 3.091 & 2.933 & 3.245 & 30 \\
Italy & Off & On & 2.965 & 2.812 & 3.126 & 30 \\
Italy & On & Off & 2.942 & 2.806 & 3.084 & 30 \\
Japan & Off & Off & 2.410 & 2.293 & 2.532 & 30 \\
Japan & Off & On & 2.472 & 2.351 & 2.599 & 30 \\
Japan & On & Off & 3.404 & 3.242 & 3.557 & 30 \\
Japan & On & On & 3.313 & 3.166 & 3.475 & 30 \\
United Kingdom & Off & Off & 2.838 & 2.673 & 3.007 & 30 \\
United Kingdom & Off & On & 3.076 & 2.884 & 3.272 & 30 \\
United Kingdom & On & Off & 3.081 & 2.915 & 3.256 & 30 \\
United Kingdom & On & On & 3.458 & 3.293 & 3.633 & 30 \\
United States & Off & Off & 3.630 & 3.451 & 3.812 & 30 \\
United States & Off & On & 3.502 & 3.310 & 3.711 & 30 \\
United States & On & Off & 3.129 & 2.969 & 3.296 & 30 \\
United States & On & On & 3.256 & 3.080 & 3.442 & 30 \\
\bottomrule
\end{tabular}
\end{table*}
\subsection{Cross-Country Consistency Diagnostics}
Table~\ref{tab:fairness} reports cross-country consistency diagnostics for
Portfolios~A and~B. The diagnostics operate on \emph{aggregated configuration-level
cells} rather than individual scenario rows. Each cell corresponds to a specific
(country, prompt-variant, RAG flag, news flag, portfolio) combination and may contain
zero, one, or multiple accepted scenarios after plausibility filtering.
At this aggregated level, coverage is complete:
$7 \times 30 \times 4 = 840$ cells per portfolio. Label flips and outliers
(defined on these cells) are rare, and cross-country VaR gaps are small
(approximately 0.03 for Portfolio~A and 0.04 for Portfolio~B in the linear
factor channel), indicating limited cross-country disparity in average tail
risk. Nonlinear-channel gaps are even smaller for Portfolio~A ($\approx0.01$),
and remain modest for Portfolio~B ($\approx0.03$).
This construction also explains why Table~\ref{tab:fairness} always
contains 840 rows per portfolio, even though the underlying deterministic
GPT-5-mini run retains fewer individual scenarios overall (e.g., 627 in
Table~\ref{tab:macro-summary}).
\begin{table*}[t]
\centering
\caption{Fairness and robustness diagnostics for Portfolios A and B.
\textbf{Computed on aggregated cells (country × prompt-variant × RAG × news × portfolio), not on individual scenario rows.}}
\label{tab:fairness}
\begin{tabular}{cccccccc}
\toprule
Portfolio &
\makecell{Rows with\\full coverage} &
\makecell{Rows with\\outcome} &
\makecell{Label flips\\under perturbation} &
\makecell{Outliers\\$z>3$} &
\makecell{Outliers\\MAD} &
\makecell{Group gap\\VaR (linear)} &
\makecell{Group gap\\VaR (nonlinear)} \\
\midrule
A & 840 & 14 & 14 & 8 & 16 & 0.033 & 0.010 \\
B & 840 & 14 & 14 & 8 & 16 & 0.040 & 0.033 \\
\bottomrule
\end{tabular}
\end{table*}
\subsection{Variance Decomposition}
Table~\ref{tab:anova-decomp} presents ANOVA results for the \emph{linear-channel}
VaR and CVaR multiples.
The ANOVA decomposition shows that portfolio identity is the single largest
driver of variance in tail risk (partial $\eta^2 \approx 0.587$ for VaR and
$0.787$ for CVaR). The choice of prompt variant is the second-largest driver
(partial $\eta^2 \approx 0.258$ for VaR and $0.261$ for CVaR), while country
effects are smaller but non-negligible (partial $\eta^2 \approx 0.046$ for VaR
and $0.043$ for CVaR).
RAG has essentially no effect in this setup ($\eta^2 \approx 0$,
$p \approx 0.59$), and news retrieval, although statistically significant,
accounts for only about 1--2\% of the variance (partial
$\eta^2 \approx 0.014$).
These findings are consistent with the qualitative patterns in
Figures~\ref{fig:cvar-by-country}, \ref{fig:news-effect}, and~\ref{fig:heatmaps}.
\begin{table}[h]
\centering
\caption{ANOVA variance decomposition of linear-channel VaR and CVaR multiples (partial $\eta^2$).}
\label{tab:anova-decomp}
\begin{tabular}{lccccc}
\toprule
Effect &
\makecell{p-val\\(VaR)} &
\makecell{$\eta^2$\\(VaR)} &
\makecell{p-val\\(CVaR)} &
\makecell{$\eta^2$\\(CVaR)} \\
\midrule
C(country) & 0.000 & 0.046 & 0.000 & 0.043 \\
C(portfolio\_id) & 0.000 & 0.587 & 0.000 & 0.787 \\
C(prompt\_variant) & 0.000 & 0.258 & 0.000 & 0.261 \\
C(rag) & 0.599 & 0.000 & 0.593 & 0.000 \\
C(use\_news) & 0.000 & 0.014 & 0.000 & 0.014 \\
\bottomrule
\end{tabular}
\end{table}
\subsection{Mean VaR Multiples by Country and Configuration}
Figure~\ref{fig:heatmaps} displays heatmaps of mean \emph{linear} VaR multiples by
country across RAG and news configurations, separated by model.
Clear cross-country heterogeneity is visible, with Japan often towards the
upper end and the U.K.\ frequently towards the lower end.
Effects of RAG/news are small but systematic, consistent with the ANOVA
decomposition and the tight confidence intervals in the bootstrap summaries.
\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{fig07_mean_risk_heatmaps_by_model.png}
\caption{Heatmaps of mean linear risk multiples for Portfolio~A by country
(rows) and configuration (columns: RAG on/off, news on/off), shown
separately for GPT-5-mini and Llama-3.1-8B-Instruct.
Darker cells indicate higher average VaR/CVaR multiples; see
Table~\ref{tab:risk-crossrun} for aggregated statistics.}
\label{fig:heatmaps}
\end{figure*}
\section{Discussion}
\label{sec:discussion}
This section interprets the empirical findings from Section~\ref{sec:results} and
draws implications for LLM-driven macro–financial stress testing.
We discuss model behavior, contextual grounding, portfolio transmission, stability risks, and ethical and operational considerations for deployment.
Throughout, we reference the empirical evidence summarised in
Tables~\ref{tab:macro-summary}--\ref{tab:anova-decomp}
and Figures~\ref{fig:macro-dists}--\ref{fig:heatmaps}.
\subsection{Model Behavior and Risk Severity}
Across both GPT-5-mini and Llama-3.1-8B-Instruct
(Table~\ref{tab:severity-model}), LLMs consistently generate contractionary GDP
shocks and moderately higher inflation and interest rates
(Table~\ref{tab:macro-summary}). Under the three-channel PCA-based translator,
these shocks propagate into VaR/CVaR multiples that are stable across
configurations and comfortably bounded relative to historical crisis episodes.
In the \emph{linear} factor channel, VaR multiples cluster around
$1.46$--$1.48\times$ for GPT-5-mini and roughly $1.41$--$1.42\times$ for
Llama-3.1-8B-Instruct. Corresponding linear CVaR multiples are about
$1.13\times$ for GPT-5-mini and $1.22$--$1.23\times$ for Llama
(Table~\ref{tab:risk-crossrun}). The \emph{pure volatility} channel produces
much larger multiples—roughly $3.6$--$3.8\times$ for VaR and $2.7$--$3.0\times$
for CVaR—reflecting inflation-driven volatility scaling of the
calm/crisis covariance mixture. The \emph{nonlinear} channel adds a modest but
non-zero increment: nonlinear CVaR multiples are about $1.07$--$1.08\times$ for
GPT-5-mini and $1.15$--$1.17\times$ for Llama, implying an additional
7--17\% uplift beyond the linear channel.
All three channels sit above the historical and econometric baselines
(Figure~\ref{fig:baselines}, Table~\ref{tab:baseline-var}) and imply stress
more severe than simple historical replay, yet remain well below the historical
crisis envelopes for Portfolio~A. Relative to the calm-period (2012--2019)
baseline, the GFC yields approximately $6.00\times$ VaR and $4.45\times$ CVaR
multiples, while COVID-19 yields $1.71\times$ and $1.12\times$, respectively
(Section~\ref{sec:crisis-envelopes}). LLM-generated linear-channel multiples in
the 1.1--1.5$\times$ band therefore resemble “moderate stress” rather than
full-blown crisis states.
These results imply that \emph{severity differences between models are small},
even though their macro shock patterns diverge modestly:
GPT-5-mini tends to produce larger interest-rate shocks, while
Llama-3.1-8B-Instruct generates slightly deeper GDP contractions
(Figure~\ref{fig:model-comparison}). GPT-5-mini exhibits consistently high
plausibility pass rates across configurations, whereas Llama shows concentrated
failures in one retrieval-and-news-enabled setting. Thus, although average
severity is similar across models, \textbf{reliability and coherence remain
model-specific}. This distinction matters for supervisory adoption:
stable behaviour under repeated prompting is at least as important as
expected severity.
\paragraph{Retrieval Dampening and the Cost--Benefit Trade-off.}
The ANOVA results in Table~\ref{tab:anova-decomp} show that retrieval has an
almost negligible quantitative effect on linear tail-risk outcomes: RAG exhibits
$\eta^2 \approx 0$, and news retrieval contributes only
$\eta^2 \approx 0.014$, whereas portfolio composition
($\eta^2 \approx 0.587$) and prompt design
($\eta^2 \approx 0.258$) dominate the variance decomposition.
This raises an important cost–benefit question regarding the retrieval
pipeline. The computational overhead of RAG is non-trivial, yet its
contribution to \emph{quantitative} scenario severity is minimal.
This behaviour is largely by design. In our architecture, the
\emph{linear} channel is deliberately insulated from text: it depends only on
numeric macro shocks and pre-estimated PCA betas, while the volatility channel
depends only on macro-derived $\lambda$ and inflation. Text, retrieval, and
news enter solely through the \emph{nonlinear} channel via a small amplification
term and polynomial betas. When an LLM outputs a shock such as
“growth $-1.4$\,pp, inflation $+3$\,pp, policy rate $+4$\,pp,” the linear
mapping distils this into low-dimensional factor shocks, and the nonlinear
channel can only perturb this mapping within capped bounds. Differences
introduced by RAG at the narrative level are therefore \emph{compressed} into
smooth, modest adjustments in nonlinear drift, which explains why RAG changes
\emph{scenario content} but only weakly affects VaR/CVaR multiples in both
linear and nonlinear channels. From a supervisory perspective, this suggests
that RAG should be viewed primarily as a \emph{narrative-grounding mechanism},
rather than a first-order driver of quantitative stress.
\subsection{Contextual Grounding and Risk Modulation}
On the linear VaR/CVaR multiples, mean levels are tightly clustered around
$1.46$--$1.48\times$ for VaR under GPT-5-mini and $1.41$--$1.42\times$ under
Llama-3.1-8B-Instruct, with corresponding CVaR multiples spanning roughly
$1.13$--$1.23\times$ across all model/RAG/news configurations
(Table~\ref{tab:risk-crossrun}). Changes in retrieval and news move these
averages by at most $0.01$--$0.02$, i.e., about 1--2\% in relative terms.
The ANOVA on these linear multiples indicates that RAG has no detectable
effect ($p \approx 0.60$, $\eta^2 \approx 0$), while news retrieval is
statistically significant but accounts for only about 1--2\% of the
explained variance ($\eta^2 \approx 0.014$; Table~\ref{tab:anova-decomp}).
Economically, the news effect is negligible: retrieval and news behave more
like mild stabilisers than first-order drivers of tail risk. This is consistent
with the fairness diagnostics, where group gaps in linear VaR multiples across
countries are on the order of $0.03$--$0.04$, and nonlinear gaps are similar or
smaller (Table~\ref{tab:fairness}).
We interpret this as follows: contextual grounding mostly limits narrative
drift rather than reshaping macro magnitudes. Several design choices help
explain why the measured effects of RAG and news are so small.
First, the retrieval corpus is intentionally narrow: country profiles are
built from the same IMF WEO template across the G7 and differ mainly in
numerical baselines and a small set of qualitative descriptors. Off-the-shelf
MiniLM embeddings are used without macro-specific fine-tuning, so the nearest
neighbours for a given country are typically economically similar peers with
overlapping content rather than qualitatively different narratives.
Second, the prompt template strongly constrains the output format and severity
range: the model is explicitly instructed to produce a “severe but plausible”
Q4--2026 stress scenario relative to the same WEO baseline, so retrieved
context acts more as a topical anchor than as a hard driver of macro shock
magnitude.
Third, only a modest amount of news information is injected into the prompt
(up to 20 clustered headlines drawn from a 50-headline snapshot), and the
macro--portfolio translator is dominated by the linear PCA mapping and
covariance mixture. Taken together, these choices make it unsurprising that
RAG and news have detectable but numerically small effects in ANOVA: they
stabilise narratives and reduce drift more than they alter the distribution of
macro shocks or tail-risk multiples. Richer retrieval corpora, macro-tuned
embeddings, or looser prompt constraints may increase retrieval impact, but at
the cost of higher variance and more challenging governance.
\subsection{Portfolio Stress and Sector Attribution}
The PCA-based translator introduces an interpretable economic mechanism:
equity-sensitive factors react primarily to growth contractions,
duration factors to rate movements, and the gold/safe-haven factor to
inflation and risk-off episodes.
This produces sizeable tail-risk amplification across asset classes and
explains the elevated VaR and CVaR multiples observed in
Figure~\ref{fig:cvar-by-country} and Table~\ref{tab:risk-crossrun}.
These patterns are relatively stable across the configurations shown in
Figures~\ref{fig:cvar-by-country} and~\ref{fig:heatmaps}, with only modest
cross-country variation in averages and gaps.
On the sector side, scenario narratives often highlight similar vulnerability
clusters—especially energy, financials, and industrials—whereas raw
LLM-generated sector labels are noisy and heterogeneous.
Canonicalisation of free-text sector references (e.g., “oil and gas producers,”
“consumer discretionary firms”) into a standard ETF taxonomy reduces spurious
heterogeneity and aligns scenario-level sector attributions with liquid traded
indexes. Under this mapping, the factor loadings derived from PCA and
regressions provide a disciplined link from text to tradable exposures.
Sector attribution is therefore one of the more robust narrative components,
provided that careful preprocessing and mapping are applied.
\subsection{Limitations and Stability Risks}
Despite strong directional performance, several key limitations remain.
\paragraph{Plausibility does not guarantee macroeconomic validity.}
The rule-based audit filters incoherent combinations such as
“recession + disinflation + rate hikes” without justification,
but nuanced macro–financial channels (e.g., currency defence,
sovereign spread dynamics, multi-country linkages) remain outside model
awareness. The DeBERTa-based regime classifier adds an additional layer by
flagging “crisis” narratives, but cannot ensure full structural consistency.
Human review is therefore essential and cannot be replaced by automated checks.
\paragraph{Reproducibility is model- and context-dependent.}
Tables~\ref{tab:prompt-dispersion} and~\ref{tab:stability-config} show that
GPT-5-mini exhibits moderate and fairly homogeneous intra-configuration
dispersion (typically 2.4--3.6 in the $(\Delta g, \Delta \pi, \Delta r)$
shock space) across countries and RAG/news settings in the deterministic run.
Residual variability arises from sensitivity to prompt wording and
context composition, rather than from explicit sampling noise.
While we do not observe extreme instabilities in this setup, the results
underscore that \textbf{retrieved context, news snapshots, and macro baselines
must be version-controlled}: deterministic decoding only guarantees stable
behaviour \emph{conditional} on a fixed context block and model version.
\paragraph{Factor model and regime-mixture limitations.}
The PCA translator and regime-mixed covariance matrix are intentionally simple.
Although the covariance matrix is scenario-specific via
$\Sigma_{\text{scen}} = (1-\lambda)\Sigma_{\text{calm}} +
\lambda\Sigma_{\text{crisis}}$, the structure is still linear in the mixture
parameter and limited to two regimes estimated from ETF returns.
This approach cannot capture several crisis dynamics that materially shape
real-world loss distributions, including (i) endogenous correlation breakdown
beyond the chosen crises, (ii) volatility clustering and jump risk, (iii)
liquidity dry-ups that amplify drawdowns, or (iv) explicit credit-spread and
funding channels. As a result, the model likely \emph{understates} tail
amplification during extreme, system-wide stress, especially when macro shocks
coincide with novel structural shifts not reflected in the calm/crisis samples.
These constraints also clarify the RAG results in
Table~\ref{tab:anova-decomp}. Retrieval and news modify the \emph{narrative}
structure of scenarios—introducing new contagion channels, geopolitical
triggers, policy constraints, or sector-specific vulnerabilities. However,
only those aspects that shift the numeric macro shocks and regime index
$\lambda$ can meaningfully change the covariance mixture or factor shocks.
Nuanced contextual information in the retrieved documents therefore does not
propagate into materially different return distributions unless it feeds back
into GDP, inflation, or policy-rate shocks, or into the regime classification.
By construction, RAG enriches \emph{scenario specificity} more than
\emph{risk multiples}, and its quantitative effects are dampened by the
linear mixture and capped nonlinear channel.
Despite these limitations, the three-channel translator is justified for this
paper’s objective: it provides a transparent, auditable, and model-agnostic
baseline that isolates the contribution of LLM-generated macro shocks and text
amplification. More sophisticated translators—such as multi-regime stochastic
volatility models, liquidity stress engines, credit-spread channels, or
multi-country VAR/SVAR systems—would introduce substantial additional model
risk, calibration choices, and opaque interactions that could obscure the role
of the LLM in shaping scenario severity. The current design therefore serves as
a \emph{controlled benchmark}: interpretable, reproducible, and suitable for
comparing LLMs, prompts, retrieval pipelines, and plausibility filters without
embedding hidden, model-specific nonlinear assumptions.
\paragraph{Regulatory deployment pathway.}
A feasible route toward supervisory use includes:
(i) complete artefact snapshotting (IMF baselines, cached prices, embeddings,
headline snapshots, PCA and covariance estimates);
(ii) comparative validation against regulator-designed scenarios and historical
episodes (e.g., matching GFC/COVID envelopes);
(iii) human-in-the-loop refinement of narratives and sector mappings; and
(iv) version-locked models, retrievers, and news feeds.
These steps align with established model risk management frameworks and ensure
auditability across the full pipeline from prompt to portfolio VaR/CVaR.
\subsection{Ethics \& Societal Impact}
The use of LLMs in stress testing raises unique risks.
\paragraph{Hallucination and overconfidence.}
LLMs occasionally invent macro linkages or misinterpret headline context.
If taken at face value, such hallucinations could distort capital planning or
hedging strategies. Grounded retrieval, plausibility audits, and
scenario-specific risk decomposition (linear vs.\ nonlinear channels) reduce
(but do not eliminate) this risk.
\paragraph{Systemic homogeneity.}
If many institutions rely on similarly tuned LLMs,
stress narratives may homogenise,
reducing scenario diversity and increasing systemic fragility.
Diverse prompt templates, retrievers, portfolios, and expert adjustments are
therefore important safeguards.
\paragraph{Data provenance and privacy.}
Our pipeline uses only public macroeconomic projections and
publicly available news headlines, with full snapshotting of sources.
No personal data are accessed or stored.
Still, institutional deployments must ensure that third-party API usage
complies with data governance rules and jurisdiction-specific privacy
requirements.
\paragraph{Bias.}
LLM outputs can reflect geopolitical and linguistic bias.
Table~\ref{tab:fairness} shows that country-level VaR gaps in both linear and
nonlinear channels are small (on the order of a few hundredths), but subtle
representational biases may persist in narratives and sector attributions.
Future work should incorporate formal bias audits, multilingual retrieval,
and human review of narrative framing.
\subsection{Implications and Future Work}
Overall, our findings indicate that LLMs can generate \emph{severe, coherent, and
broadly stable} macro–financial scenarios when properly grounded and audited.
We highlight four areas for future development:
\begin{enumerate}
\item \textbf{Scenario evaluation.}
We propose a composite evaluation suite combining
(i) plausibility audits and regime scores,
(ii) intra-prompt and intra-configuration dispersion metrics, and
(iii) econometric anchors and crisis envelopes (e.g., EWMA, GARCH, GFC/COVID
baselines). Standardising such diagnostics would advance the field beyond
ad hoc scenario grading.
\item \textbf{Behavioral alignment.}
Reinforcement learning from expert judgment, or light finetuning on
historical crisis narratives, may reduce variance and improve macro–logical
consistency, especially around edge cases such as policy-constraint regimes
and multi-country contagion.
\item \textbf{Human oversight tools.}
Practical deployment requires interfaces for narrative review,
sector attribution checks, and editable scenario components, together with
channel-wise risk decomposition (volatility vs.\ linear vs.\ nonlinear).
LLMs should act as scenario \emph{assistants}, not autonomous generators.
\item \textbf{Scalability and extension.}
The pipeline parallelises across countries and portfolios.
FAISS indexing and MiniLM embeddings scale efficiently; risk simulation and
covariance estimation remain the dominant compute costs.
Extending to richer factor structures, credit and FX channels, and
multi-country balance-sheet data is a natural next step, provided that
additional model complexity is paired with commensurate governance.
\end{enumerate}
LLM-driven stress testing is feasible and powerful,
but requires strong grounding, explicit separation of linear and nonlinear
channels, reproducibility scaffolding, and human oversight.
With these controls, LLMs can meaningfully augment the generation and evaluation
of supervisory macro–financial scenarios.
\section{Conclusion}
\label{sec:conclusion}
This paper presented a hybrid LLM-based pipeline for macro--financial stress
testing that couples institutional baselines with retrieval-augmented large
language models and deterministic, LLM-free risk baselines.
Country-specific macro fundamentals from the IMF
\textit{World Economic Outlook}, optionally enriched with recent news, are
embedded with MiniLM and indexed via FAISS to provide semantically grounded
context. GPT-5-mini and Llama-3.1-8B-Instruct then generate structured,
machine-readable scenarios---JSON shocks to GDP growth, inflation, and
interest rates, plus narrative rationales and sector tags---for a common
Q4~2026 horizon, subject to a two-layer plausibility audit and a
DeBERTa-based regime classifier.
We map these shocks into tradable portfolios through a three-channel,
PCA-based translator anchored in historical ETF returns and regime-specific
covariance matrices. A pure volatility channel scales a calm/crisis covariance
mixture as a function of inflation and regime severity; a linear channel
propagates macro shocks into factor drifts via transparent PCA betas; and a
nonlinear channel applies capped polynomial betas with modest amplification
from text, retrieval, and news. Relative to historical and econometric
baselines, the resulting LLM scenarios produce **moderate but material
tail-risk amplification**. In the linear channel, VaR multiples concentrate
around $1.46$--$1.48\times$ for GPT-5-mini and $1.41$--$1.42\times$ for
Llama-3.1-8B-Instruct, while CVaR multiples lie roughly between $1.13$ and
$1.23\times$ (with GPT-5-mini near the lower end and Llama near the upper end),
with small standard deviations across scenarios
(Table~\ref{tab:risk-crossrun}, Figure~\ref{fig:cvar-by-country}).
The volatility channel yields VaR and CVaR multiples around
$3.6$--$3.8\times$ and $2.7$--$3.0\times$, respectively, while the nonlinear
channel adds an incremental 7--17\% uplift over the linear channel.
All of these lie well below the historical GFC envelopes
(roughly $6\times$ VaR and $4.5\times$ CVaR relative to a calm baseline;
Section~\ref{sec:crisis-envelopes}), indicating that LLM-induced stress in this
setup corresponds to “moderate stress” rather than full crisis calibration.
Retrieval grounding and contemporaneous news act as **directional stabilisers**
rather than first-order drivers of tail risk.
On the linear VaR/CVaR multiples, mean levels are tightly clustered
around $1.46$--$1.48$ for VaR under GPT-5-mini and $1.41$--$1.42$ under
Llama-3.1-8B-Instruct, with CVaR multiples spanning roughly
$1.13$--$1.23\times$ across all model/RAG/news configurations
(Table~\ref{tab:risk-crossrun}), and turning retrieval or news on or off
moves these averages by at most about 1--2\% in relative terms.
The ANOVA confirms that RAG has no detectable effect on linear-channel
multiples, while news retrieval is statistically significant but explains only
around 1--2\% of the variance (Table~\ref{tab:anova-decomp}), so its economic
impact on tail risk is small.
Cross-run comparisons show that GPT-5-mini and Llama-3.1-8B-Instruct deliver
broadly similar portfolio-level risk multiples, suggesting that portfolio
composition and macro shock design matter more than the specific LLM choice
for tail-risk levels in this setup.
On the governance side, we introduced a set of diagnostics tailored to
regulatory use.
A rule-based macro plausibility audit, combined with a soft plausibility score
and regime classification, filters out incoherent combinations of growth,
inflation, interest rates, and narrative rationales, yielding high pass rates
across configurations.
A reproducibility lens based on intra-prompt and intra-configuration dispersion
shows that GPT-5-mini is consistently stable under deterministic prompting,
with dispersion typically between 2.4 and 3.6 in the
$(\Delta g, \Delta \pi, \Delta r)$ shock space
(Tables~\ref{tab:prompt-dispersion} and~\ref{tab:stability-config}).
Fairness cards built on aggregated configuration-level cells
(Table~\ref{tab:fairness}) indicate complete coverage, rare label flips under
perturbations, and small cross-country group gaps in both linear and nonlinear
VaR multiples (on the order of a few hundredths), suggesting that no single
country is systematically favoured or penalised in tail-risk metrics.
These findings underscore that **deterministic decoding alone is
insufficient**: stability and fairness depend on version-controlled retrieval
indices, news snapshots, macro baselines, and prompts, together with explicit,
channel-wise risk decomposition.
The study also surfaces practical limitations.
Sector attributions require canonicalisation before mapping to tradable ETFs;
the factor and covariance structure focus on equity, duration, and gold
channels and omit explicit policy feedbacks, credit spreads, contagion, and
liquidity spirals; and plausibility rules plus NLI-based regime tags are not a
substitute for expert macroeconomic judgment.
Nonetheless, the overall picture is encouraging: with structured prompts,
retrieval grounding, plausibility checks, dispersion diagnostics, crisis
envelopes, and snapshotting, LLM pipelines can complement traditional
supervisory stress testing by generating country-specific,
auditable macro shocks that are both quantitatively anchored and
narratively rich.
Future work should extend this framework along three axes:
(i) richer macro–financial translators (e.g., multi-regime stochastic
volatility, credit and FX channels, structural VARs, or multi-asset factor
systems) while preserving interpretability;
(ii) behavioural alignment using expert feedback or light finetuning to further
stabilise and de-bias narratives, especially in policy-constraint and
contagion regimes; and
(iii) human-in-the-loop interfaces for reviewing, editing, and approving
LLM-generated scenarios, with clear separation of volatility, linear, and
nonlinear channels.
With these safeguards in place, LLMs can evolve from experimental tools
into responsible co-pilots for macro--financial scenario design.
\begin{acks}
This research was conducted independently and received no financial support from any institution, organization, or funding agency. The author is solely responsible for the content and conclusions of this work.
\end{acks}
\bibliographystyle{ACM-Reference-Format}
\bibliography{sample-base}
\clearpage