Transformer-based CoVaR: Systemic Risk in Textual Information

Transformer-based CoVaR: Systemic Risk in Textual Information

\footnotetext[1]{ Department of Economics, Econometrics and Finance, Faculty of Economics and Business, University of Groningen, The Netherlands. Emails: [email removed], [email removed], [email removed]. }

\footnotetext[2]{ Corresponding author. School of Economics, University of Bristol, UK. Address: Beacon House, Queens Road, Bristol BS8 1QU, UK. Email: [email removed]. Phone: +44 7413 466969. }

Introduction

Between 2007 and 2009, the crisis in the U.S. mortgage market set off a chain reaction that destabilized financial institutions and the global economy. To better anticipate these events, regulators, policymakers, and researchers focused their attention on measuring the systemic risk that arises in a strongly connected economy. While some systemic risk factors can be effectively distilled from numerical market data, additional information can be extracted from textual data, one well-known example being “market sentiment” 2017_Keiber_and_Samyschew. However, this text-based risk information is inherently difficult to quantify, and a key open question is how we can effectively extract it from textual sources to obtain better measures and forecasts of systemic risk.

In this paper, we propose a formal econometric framework that uses a Transformer-based architecture 2017_Vaswani_et_al to improve systemic risk analysis by integrating structured financial variables and unstructured textual information such as financial news articles. Our approach proceeds in two steps. First, we employ a large language embedding model that maps variable-length news articles into fixed-size, high-dimensional vectors representing semantic content. Second, we extract the key latent information from these high-dimensional input sequences by leveraging the Transformer's ability to capture nonlinear dependencies and contextual relationships across diverse data sources (i.e., multimodal data\footnote{By multimodal data, we mean inputs of different types, combining structured numerical variables (e.g., returns) with unstructured information such as text.}).

A Transformer is a neural network that relies on the self-attention mechanism to compute latent representations of its input. Traditional neural network architectures (such as convolutional or recurrent networks) rely on fixed parameterized transformations of their inputs, whereas the self-attention mechanism in a Transformer dynamically adapts the transformation based on the input. In other words, self-attention allows the importance of a variable to change based on the surrounding data. For example, a negative news item may be weighted as highly significant for predicting risk when other variables mention "default", whereas the same news item might be down-weighted if the context is a "holiday". By identifying these context-dependent relationships, the Transformer provides the latent representations necessary for the risk prediction task.

A main motivation for using the Transformer is that traditional sentiment scores often miss critical information in textual data or may even misrepresent sentiment indicators relevant for market prediction 2019_Ke_et_al, 2011_Loughran_and_McDonald, 2007_Tetlock. Transformers can effectively learn signals from noisy, high-dimensional textual inputs by down-weighting irrelevant content 2022_Edelman_et_al, even when the sample size is limited 2025_Mousavi-Hosseini_et_al. To show that such results can be also be expected when estimating the CoVaR on relatively small financial data sets, we establish a new convergence result that complements recent theory on convergence rates for deep neural networks in high dimensions 2020_Schmidt-Hieber, 2019_Suzuki. Specifically, we show how the quantile regression framework of 2022_Padilla_et_al can be extended to incorporate text information through the Transformer architecture. We then show that under the architecture we propose, the convergence rate of the CoVaR estimator only depends logarithmically on the dimension of the input data. As a result, a researcher does not manually need to summarize unstructured text into a low-dimensional object, for example via a sentiment scores, and then specify how these summaries affect risk.

Empirically, we estimate the Conditional Value-at-Risk (CoVaR), a widely used measure of systemic risk, for eight Global Systemically Important Banks (G-SIBs) in the U.S. We consider daily returns between 2006--2013, including both the 2008 financial crisis and the 2011 debt ceiling crisis. We augment the standard numerical data used to estimate CoVaR with a rich set of Reuters financial news articles. The Transformer-based method that incorporates the full text data yield more negative CoVaR and $\Delta$CoVaR values during crisis periods than approaches that either exclude news information or reduce it to sentiment scores prior to risk modeling. After addressing potential look-ahead bias, we show that our empirical results remain robust. Furthermore, we demonstrate that incorporating text into the Transformer architecture achieves generally lower prediction loss compared to the model relying only on structured financial return data.

In summary, this paper makes three contributions to advancing the understanding and practical application of multimodal data in systemic risk assessment. First, to our knowledge, we are the first to introduce an econometric framework for analyzing CoVaR using the Transformer-based architecture. Second, we establish upper bounds for its $\ell_2$ risk and also generalization error. Finally, we empirically demonstrate that incorporating textual information leads to better predictive performance and alters the CoVaR risk measures. In particular, we identify a pronounced negative dip during the crisis period across several equity assets when comparing the Transformer-based CoVaR to both the CoVaR without text and the CoVaR using the traditional sentiment measure.

Outline. The remainder of the paper is organized as follows. Section (ref) reviews the related literature. Section (ref) introduces the model and estimators. Section (ref) presents the main theoretical results. Section (ref) reports the findings of the empirical application. Section (ref) concludes and discusses directions for future research. Appendix (ref) provides the description of an alternative Transformer architecture. Appendix (ref) discusses the interaction between news content and risk prediction. Appendix (ref) provides additional results from the simulation study. Proofs of all theoretical results are provided in the Supplemental Appendix (SA) to streamline the main exposition; these results may also be of independent technical interest.

Literature Review

Broadly speaking, we can distinguish three approaches to risk modeling in a connected world which we refer to as network-based models, simulation-based approaches and market-based approaches. Early network-based models 2000_Allen_and_Gale, 2012_Billio_et_al, 2014_Diebold_and_Yilmaz, 2015_Acemoglu_et_al highlight the importance of capturing interconnectedness and nonlinear contagion effects. Simulation-based approaches 2011_Gai_et_al, 2015_Greenwood_et_al underscore amplification mechanisms like fire sales and liquidity spirals, yet they often assume simplified dynamics.

Market-based approaches offer another perspective by deriving systemic risk measures from asset prices and market information: 2017_Acharya_et_al proposed Systemic Expected Shortfall (SES) and its variant to estimate capital shortfalls in systemic crises, and 2009_Segoviano_and_Goodhart developed the Distress Insurance Premium (DIP) using option-implied risk-neutral probabilities. Among market-based approaches, the CoVaR framework proposed by 2016_Adrian_and_Brunnermeier remains a central tool for quantifying systemic risk. Subsequent contributions, including 2015_Hautsch_et_al, 2016_Haerdle_et_al, and 2022_Keilbar_and_Wang, extend CoVaR to high-dimensional environments. The high-dimensional setting is essential in applied practice, as systemic risk emerges from the complex interconnectedness of global markets and macroeconomic factors that low-dimensional models fail to address simultaneously. However, these methodologies do not accommodate multimodal data structures, which are increasingly relevant in modern financial systems.

A popular strategy to accommodate information from text is the use of sentiment analysis, where texts are for example classified as positive, negative, or neutral. Early work constructs sentiment indices using dictionary-based word counts: 2007_Tetlock shows that pessimistic media tone predicts short-term market declines, while 2011_Loughran_and_McDonald refine sentiment lexica to better capture financial context. Later studies, such as 2019_Ke_et_al, employ topic modeling to assign sentiment weights to terms and aggregate them into document-level scores for return prediction. Even recent neural-network-based studies are sentiment-focused: 2024_Schuettler_et_al find that LLMs enhance stock return prediction, 2024_Wang_and_Ma propose MANA-Net to aggregate sentiment dynamically by market relevance, and 2025_Jha_et_al measure popular financial sentiment across languages using LLMs. These findings highlight the value of richer linguistic and contextual signals through market sentiment. However, reducing text to low-dimensional sentiment categories may overlook other dimensions of information.

Much of the literature has found evidence of other channels affects financial risk and market outcomes beyond sentiment. For example, linguistic characteristics such as grammatical structure, linguistic complexity, etc. affect how information is processed and priced by markets 2014_Loughran_and_McDonald, 2024_Fedyk, 2024_AlexKim_et_al. In particular, 2011_Engelberg_and_Parsons and 2012_Dougal_et_al establish causal links between differential reporting of events by media or journalists and financial markets. Beyond linguistic characteristics, recent studies demonstrate the informational value of textual data for asset pricing and financial risk (e.g., 2025_Adammer_et_al; 2023_van_Dijk_and_de_Winter; 2020_Bybee_et_al), consistent with psychological evidence linking emotions to financial decisions 2000_Lerner_and_Keltner, 2001_Lerner_and_Keltner. In addition, topic-based analyses such as 2009_Kogan_et_al and 2019_Larsen_and_Thorsrud show that latent news topics predict volatility and macroeconomic variables. Thus, focusing solely on simplified sentiment analysis may be insufficient to capture the dynamics of risk, particularly during periods of financial stress.

Recent advances in multimodal learning reinforce the view that textual information can provide valuable predictive signals for financial markets beyond traditional sentiment indicators. Transformers and LLMs have been applied to financial prediction using diverse inputs, including earnings-call audio, social media, and news 2020_Li_et_al, 2023_Zou_and_Herremans, 2024_Wang_et_al, 2024_Omranpour_et_al, 2025_Wang_et_al, 2020_Yang_et_al, 2025_Cao_et_al. Beyond multimodal prediction, 2026_Pele_et_al examine LLMs guided by human prompts for financial forecasting. Our empirical application builds on these approaches by employing a large language embedding model that maps news articles into high-dimensional vectors representing semantic content, rather than collapsing them into a single sentiment score.

Methodology

In this section, we first introduce the systemic risk modeling framework and then the Transformer-based architecture used for approximating a quantile function that integrates both financial variables and market-related textual information. In the next section, we formally study its approximation and generalization properties.

Systemic Risk Modeling

Our CoVaR estimation procedure proceeds in two steps, which builds on the framework of 2022_Keilbar_and_Wang but modifies the second estimation step. In the first step, we estimate the VaR for each bank using a linear quantile regression model. In the second step, we estimate CoVaR by first fitting a Transformer-based quantile regression conditional on other banks’ returns and all financial news, and then plugging in the corresponding VaR estimates from the first step to replace the returns. We do not incorporate text data in the first step to maintain the standard definition of VaR as a quantile of the marginal return distribution. Our focus is therefore on the second step, where we use the Transformer's ability to process textual data.

In the following, we let $R_{j,t}\in\mathbb{R}$ denote the return of bank $j$ at time $t$, where $j=1,\ldots,J$ with $J$ fixed and finite, and $t=1,\ldots,T$. To incorporate textual information, we utilize an embedding mapping, which is referred to in the deep learning literature as an embedding layer. This embedding layer maps individual news items into $d_e$--dimensional numerical vectors (i.e., embedding vectors) that represent their semantic content in a continuous coordinate space. Specifically, let $n$ denote the number of news items observed for bank $j$ at time $t$. Applying the embedding layer to each news item yields a news-embedding matrix $E_{j,t} \in \mathbb{R}^{d_e \times n}$, where each column represents a single news item and $d_e$ is the embedding dimension. In our empirical application (see Section (ref)), we employ a pre-trained\footnote{A pre-trained model is trained in advance on large, general data.} mapping $\pi_e$ and treat its parameters as fixed. In practice, one may instead fine-tune\footnote{Fine-tuning means starting from a pre-trained model and then slightly updating its parameters using task-specific data.} its embedding parameters.

Step 1: Estimation of VaR\\ A variety of methods has been developed to estimate VaR, including the historical simulation method 1999_Hull_and_White, parametric volatility models such as ARCH and GARCH 1982_Engle_et_al,1986_Bollerslev, approaches based on Extreme Value Theory 2000_McNeil_and_Frey, and linear quantile regression 2015_Chao_et_al. Any of the approaches described above can be used to construct our VaR estimator. Among these approaches, linear quantile regression provides a transparent, flexible, and easily interpretable alternative 1978_Koenker. It has been widely used in empirical studies linking macro-financial conditions to tail risks 2016_Adrian_and_Brunnermeier, 2016_Haerdle_et_al. Following 2016_Adrian_and_Brunnermeier, let $M_{t-1} \in \mathbb{R}^{m}$ denote a vector of lagged macroeconomic state variables that reflect the general state of the economy. The VaR for bank $i$ at time $t$ is defined as the $\tau$-quantile of its return distribution conditional on $M_{t-1}$:

By convention, $\operatorname{VaR}_{i,t}^{\tau}$ is typically negative, reflecting a potential loss. Although alternative sign conventions exist, we follow the one stated above. We employ a linear quantile regression with lagged macroeconomic variables to estimate $\operatorname{VaR}_{i,t}^{\tau}$:

with $\alpha_{i}(\tau) \in \mathbb{R}$, $\gamma_{i}(\tau)\in \mathbb{R}^{m}$. We assume $F^{-1}_{\epsilon_{i,t}(\tau) \mid M_{t-1}}(\tau) = 0$ with $\epsilon_{i,t}(\tau)$ being the error term for the $\tau$-th quantile. The VaR estimate is then the fitted value of the quantile regression

Step 2: Estimation of CoVaR\\ CoVaR, introduced as a systemic extension of standard VaR by 2016_Adrian_and_Brunnermeier, is defined as the VaR of bank $j$ conditional on a specific state of the financial system. In particular, we consider the specific state as the distress scenario of the financial system, while also taking the market-related textual information (i.e., financial news) into account. For the distress scenario of financial systems, we assume that all other banks are at their VaR level. Formally, we define CoVaR as

where $R_{-j,t}, \operatorname{VaR}_{-j,t}^{\tau} \in \mathbb{R}^{J-1}$ denote the returns and VaRs of all banks other than $j$ at time period $t$, respectively, and {$E_{j,t}$ relates to the embedded information for $j$-th bank up to at most time period $t$ (e.g., embedded financial news at time period $t-1$)}. We estimate $\operatorname{CoVaR}_{j,t}^{\tau}$ using a Transformer-based quantile regression described in Section (ref). Specifically, the model is given by

where $f_{j,\tau}^*(\cdot)$ is the true conditional quantile function for bank $j$, and $\nu_{j,t}$ is an error term satisfying $F^{-1}_{\nu_{j,t}(\tau) \mid R_{-j,t}, E_{j,t}}(\tau) = 0$. The function $f_{j,\tau}^*$ is essential for capturing the complex tail behavior of returns and news. It serves as the channel through which non-sentiment-related risk, as introduced earlier, transmits to the target bank return. Under the distress scenario, we replace $R_{-j,t}$ with the VaR estimates of all other banks $\widehat{\operatorname{VaR}}_{-j,t}^{\tau}$, so that the estimated CoVaR is computed as

where $\widehat{f}_{j,\tau} \in \mathcal{G}$, with $\mathcal{G}$ denoting a class of Transformer-based neural network functions. The class $\mathcal{G}$, along with the model architecture and estimation procedure, is introduced in the next section.

Transformer Architecture

We adopt the Transformer architecture for the following reasons. Firstly, a key challenge for our CoVaR framework is integrating numerical and textual inputs without imposing restrictive dependence assumptions. Conventional architectures like CNNs (assuming spatial locality) and recurrent architectures like RNNs (assuming temporal order) 2017_Yin_et_al embed strong functional constraints that may not hold for our data. In contrast, the Transformer architecture enables direct pairwise interactions among all input elements, thereby capturing global dependencies 2017_Vaswani_et_al, 2020_Cordonnier_et_al. This may make the Transformer architecture better suited to settings where neither specific spatial nor temporal structure is clearly known a priori.

Secondly, our application necessitates a model that can learn sparse functions from noisy data. Financial news contains significant redundancy and irrelevance, requiring the model to identify and focus on meaningful information and signals. Additionally, the volume of news varies daily, requiring the use of padding\footnote{Padding refers to extending shorter sequences with special meaningless placeholders so that all sequences have the same length. This is necessary because neural networks typically require inputs of uniform size.} to create uniform input sequences. This introduces artificial tokens\footnote{In Transformers, a token is the basic unit of text that the model processes. It is analogous to a variable or feature in an econometric dataset. A token is typically represented as a vector. Padding tokens act as placeholders.} that the model must disregard. The Transformer’s self-attention mechanism directly addresses both requirements. It enables the model to dynamically weight input importance, effectively ignoring both irrelevant content and padding tokens. This property aligns with its theoretically established capacity to learn sparse functions (2022_Edelman_et_al).

Thirdly, another challenge stems from the high-dimensional input, characterized by a large number of news articles $n$ (i.e., the model input length or number of tokens) and a large embedding dimensions $d_e$. Recent theoretical work suggests this challenge can be mitigated. 2024_Trauger_and_Tewari demonstrate that, under suitable parameter constraints, the complexity of a Transformer model can even grow independently of the input dimension. In addition, 2025_Mousavi-Hosseini_et_al shows that Transformers can have smaller sample complexity that almost independent of the input length, implying superior data efficiency compared to feedforward or recurrent architectures. Inspired by these insights, we adopt a function class composed of Transformer blocks where the parameter norms are constrained. This design ensures that the model's complexity—and consequently, the input dimensions can have little effect on the model complexity, as demonstrated in Corollary (ref).

The Transformer architecture contains two components that we discuss in depth below: a multi-head self-attention mechanism (MSA) and a position-wise feed-forward network (FFN). We refer to these two components as sublayers. Each sublayer can be wrapped with a residual (skip) connection and layer normalization 2016_He_et_al, 2016_Ba_Kiros_et_al, which stabilize training and mitigate issues such as vanishing gradients. One MSA sublayer and one FFN sublayer together form a Transformer layer. A Transformer model can contain $L$ such layers stacked sequentially, indexed by $l=1, \ldots, L$. In the following, we denote the rectified linear unit (ReLU) activation function by $\sigma_{R}(x)=\max (x, \ 0)$ which operates element-wise, regardless of whether the input is a vector or a matrix. We also define the SoftMax function as

We now zoom in on the Transformer architecture. We consider the Transformer architecture without residual connections and layer normalization in this paper. For a discussion on architectures including these components, please refer to 2025_Jiao_et_al. While these operations provide additional transformations within each sublayer, they do not alter the fundamental input–output mapping structure of the Transformer block 2022_Edelman_et_al. For completeness, we present in Appendix (ref) the standard Transformer architecture, including residual connections and layer normalization.

We take the initial input to our Transformer-based neural network to be a matrix $Z_{j,t}\in\mathbb{R}^{d\times n}$ formed by concatenating the returns $R_{-j,t}$ and the text embeddings $E_{j,t}$. In practice, $R_{-j,t}$ and $E_{j,t}$ may not have compatible dimensions, so we apply a mapping $\Pi$ to obtain a representation of the desired size. In the empirical application in Section (ref), we consider a function $\Pi$ of the following form:

where $\mathbf{1}_{n} \in \mathbb{R}^{n}$ is a vector of ones and $d = (J-1)+d_e$. This construction replicates the same return vector $R_{-j,t}$ across the $n$ columns and stacks it above each column of $E_{j,t}$. We refer to the resulting columns as return-augmented news embeddings.

Self-Attention Sublayer

The self-attention sublayer allows each position (or token) in the input sequence to attend to, and aggregate information from, all other positions in the same sequence 2017_Vaswani_et_al. In our application, the input sequence is the concatenated vector of embedded news items and contemporaneous returns from other banks (see Equation (ref)). The self-attention mechanism computes pairwise attention weights between all elements in this sequence. Intuitively, these weights act as similarity or correlation scores, enabling the model to determine how much focus to place on a given news article when processing another (see Figure (ref)). A weighted sum of the input values, using these attention weights, produces the sublayer's output, blending relevant information from across the entire input context.

Multi-head self-attention extends the above mentioned idea by splitting the model dimensions $d$ into multiple heads. Let the number of heads be $H$. We define the $l$-th multi-head self-attention class $\mathcal{G}_{l}^{(MSA)}(n,d,H)$ as the set of map $g_l^{(msa)}: \mathbb{R}^{d \times n} \to \mathbb{R}^{d \times n}$ of the form

For each head $h \in \{1, \ldots, H\}$, we denote by $W_{l, h}^{(V)}, W_{l, h}^{(K)}, W_{l, h}^{(Q)} \in \mathbb{R}^{\frac{d}{H} \times d}$, and ${W_{l,h}^{(O)}} \in \mathbb{R}^{d \times \frac{d}{H}}$ the value, key, query, and projection matrices, respectively. Both the query and key matrices are learned from data during training as weighted transformations of the input matrix $Z$. The term $Z^{\top} W_{l,h}^{(K)\top} W_{l,h}^{(Q)} Z$ captures pairwise interactions between all columns of $Z$, measuring how similar or relevant one observation (e.g., news item) is to another. Estimating \( W_{l,h}^{(K)\top} \) and \( W_{l,h}^{(Q)} \) helps us understand which inputs contribute to the risk of bank \( j \). For example, if there is a strong correlation between News 3 and News 4, the corresponding weights will reflect the importance of both components in determining the bank’s risk (Figure (ref)). As another illustration, News 2 and News 3 exhibit weaker interactions relative to other pairs in Figure (ref). After dividing by $\sqrt{d}$ and applying the softmax function to normalize the weights, the self-attention mechanism amplifies strong dependencies and filters out weak or noisy signals.

In our risk analysis framework, the input matrix $Z_{j,t,s}$ jointly incorporates numerical return information and textual news content. By concatenating returns with news embeddings, we construct return-augmented news representations. The self-attention mechanism then learns the interactions among these return-augmented news inputs, allowing cross-bank spillovers and textual information to be modeled in a unified manner. In particular, the model identifies which latent information in bank returns and news articles the strongest influence on the risk exposure of firm $j$. Through this mechanism, the model uncovers dynamic transmission channels through which financial and informational shocks propagate across the system.

Feed-Forward Network Sublayer

Following the attention sublayer, a feed-forward network (FFN) sublayer applies the same operation to each column (i.e., token) of the input matrix. Its primary role is to provide additional non-linearity and transformation capacity. Recent work by 2023_Sonkar_and_Baraniuk shows that the FFN prevents the output from collapsing into a narrow subspace of the total embedding space, meaning they become highly correlated and lose their distinctiveness.

The $l$-th feed-forward network class $\mathcal{G}_{l}^{(FF)}(d, d_h)$ is defined as the set of maps $g_l^{(ff)}:\mathbb{R}^{d \times n} \to \mathbb{R}^{d \times n}$ of the form

where $d_h$ denotes the hidden dimension of the feed-forward network. The parameters are $W_l^{(1)} \in \mathbb{R}^{d_h \times d}$, $b_l^{(1)} \in \mathbb{R}^{d_h}$ and $W_l^{(2)} \in \mathbb{R}^{d \times d_h}$, $b_l^{(2)} \in \mathbb{R}^{d}$.

Transformer-based Estimator

We employ a Transformer-based estimator consisting of the previously introduced Transformer architecture followed by a multilayer perception (MLP). To bridge the Transformer output matrix $Z \in \mathbb{R}^{d \times n}$ and the MLP, we apply a linear projection. The linear projection maps the Transformer output onto a vector, which is then passed through the MLP to produce the scalar result. As shown in Figure (ref), our estimator first lets the MSA learn the interactions among the return-augmented embeddings, which yields hidden representations that are then passed through the FFN and the MLP to produce the risk prediction.

Formally, we define the Transformer-block function class $\mathcal{G}^{(TF)}(n, d, H, d_h, L)$ as the set of maps $g^{(tf)}: \mathbb{R}^{d \times n} \to \mathbb{R}^{d \times n}$ formed by the composition of $L$ Transformer layers. Each layer consists of a multi-head self-attention sublayer and a feed-forward network sublayer:

And we define the MLP function class $\mathcal{G}^{(MLP)}(D, \widetilde{d}_{m})$ as the set of maps $g^{(mlp)}: \mathbb{R}^d \to \mathbb{R}$ of the form

where the dots denote repeated compositions across layers, $D$ is the MLP network depth and $\widetilde{d}_{m}:=\left(d, d_{m}^{(1)}, \ldots, d_{m}^{(D-1)}, 1\right)$ represents the MLP network width\footnote{Depth refers to the number of layers in the MLP, and width indicates the number of neurons within each of those layers.}. If all hidden layers have the same width, we denote it by $d_m$. For $i=1, \ldots, D$, we have $W_i$ as a $d_{m}^{(i+1)} \times d_{m}^{(i)}$ weight matrix and $b_i$ as a vector of dimension $d_{m}^{(i)}$. Thus, the full class of our Transformer-based neural network functions of a given architecture $(n, d, H, d_h, L, D, \widetilde{d}_{m})$ takes the following form:

For convenience, we additionally define for a given concatenation function $\Pi$ (see, e.g., equation ((ref))),

so that for any $f \in \mathcal{F}(n, d, H, d_h, L, D, \widetilde{d}_{m})$, there exists $g \in \mathcal{G}(n, d, H, d_h, L, D, \widetilde{d}_{m})$ such that

The Transformer-based quantile estimator is the solution to the following minimization problem for a given class $\mathcal{F}$:

We apply the stochastic gradient descent (SGD) algorithm, see, e.g., 2022_Garg_et_al, 2023_Bai_et_al, 2024_Li_et_al, for solving the optimization problem in the later applications.

In summary, we construct a Transformer-based quantile regression model for CoVaR that combines peer returns and financial news within a unified architecture (see Figure (ref)). By explicitly defining the model architecture and its associated function class, we establish a foundation for analyzing both the theoretical properties and empirical performance of the proposed estimator. {In the following section, we introduce the consistency property of our CoVaR estimator.}

Theoretical Results

Notation. We consider the following notation. Let $\mathbb{R}^{+} := \{x \in \mathbb{R} : x > 0\}$ denote the set of positive real numbers. For vectors, $\|\cdot\|_{p}$ denotes the $\ell_p$ norm, and $\|\cdot\|_\infty := \max_{1 \le i \le d} |\cdot|$ is the sup-norm. For a matrix $A \in \mathbb{R}^{a \times b}$, $\|\cdot\|_{2}$ denotes the spectral norm. Let $\|\cdot\|_{p, q}$ denote the ($p, q$) matrix norm defined by first taking the $p$-norm of each row of the matrix, and then taking the $q$-norm of the resulting vector. We also define $\|A\|_0 = \left|\left((i, j): A_{i, j} \neq 0\right)\right|$ and $\|A\|_{\infty} := \max _{i=1, \ldots, p, j=1, \ldots, q}\left|A_{i, j}\right|$. For two sequences, $\{a_n\}_n$ and $\{b_n\}_n$, we write $a_n \lesssim b_n$ if there exists a constant $C$ such that $a_n \leq C b_n$ for all $n$. Moreover, for deterministic sequences $\{a_n\}_n$ and $\{b_n\}_n$ with $b_n>0$, we write $a_n = O(b_n)$ if there exists a constant $C>0$ such that $|a_n| \le C b_n$ for all sufficiently large $n$. For random variables $\{X_n\}_n$ and deterministic $\{b_n\}_n$, we write $X_n = O_p(b_n)$ if $X_n/b_n$ is bounded in probability, and write $X_n \lesssim_P b_n$ if $X_n = O_p(b_n)$.

Define the functional class $V_{(\mathcal{F}, \|\cdot\|_{\infty})}(\delta)$ as the logarithm of the $\delta$-covering number of a function class $\mathcal{F}$ under the sup-norm, which measures the complexity of $\mathcal{F}$ by counting the minimum number of balls of radius $\delta$ needed to cover it. Specifically, it is defined as $V_{(\mathcal{F}, \|\cdot\|_{\infty})}(\delta) := \log N_{\infty}(\delta,\mathcal{F})$, where $N$ is the smallest number of functions $f_1, \dots, f_N$ such that every $f \in \mathcal{F}$ is within $\delta$ (in sup-norm) of some $f_i$.

Our main theorem is in the fashion of 2019_Suzuki, 2020_Schmidt-Hieber and 2022_Padilla_et_al but we adapt these results to the CoVaR framework for Transformers. We first outline the required assumptions.

The above CDF Boundedness assumption ensures that the conditional distribution of returns is sufficiently smooth and strictly increasing near the quantile of interest 2022_Padilla_et_al. This allows well-defined CoVaR values and ensures that risk estimates respond meaningfully to fluctuations in the underlying data.

The second part in Assumption (ref) is trivial satisfied if we impose norm bounds over parameters used in $\mathcal{F}$, as the class $\mathcal{F}$ consists of functions $f$ formed by composing layers that are constructed via linear maps and Lipschitz nonlinear activations/functions. By imposing norm bounds on the parameters of each layer, we ensure each layer is Lipschitz-continuous. Consequently, the overall Lipschitz constant $L_{\mathcal{F}}$ for any $f \in \mathcal{F}$ is bounded by the product of the layer-wise constants. See Appendix (ref) for an explicit example where we consider a one-layer Transformer followed by an MLP with spectral norm bound $B$.

Under the Lipschitz Continuity and VaR Consistency assumptions, the sampling errors from VaR estimation vanish as the sample size increases, provided the overall Lipschitz constant $L_{\mathcal{F}}$ is of sufficiently small magnitude. This guarantees that the "plug-in" step does not introduce persistent bias, allowing the CoVaR estimator to remain stable and accurate 2019_Patton_et_al, 2025_Dimitriadis_and_Hoga.

Theorem (ref) connects the complexity of the estimator to an upper bound on the generalization error of our estimator. In this sense, it is comparable to Theorem 2.6 in 2020_Hayakawa_and_Suzuki, but is established here under a quantile-loss optimization framework. The result is useful for deriving convergence rates once we impose structure on both the target function and the function class $\mathcal{F}$ used for fitting. Corollary (ref) further shows that, under explicit sparsity and smoothness restrictions on the target function, one can obtain a rate whose dependence on the input dimension $d$ and input length $n$ is at most logarithmic.

Although we acknowledge that this rate stated in Corollary (ref) can be further sharpened, it already indicates that a plausible approximation remains feasible even with relatively high-dimensional input. We defer the full proof to Appendices (ref) and (ref), but first briefly summarize the main mechanisms that make such a rate attainable.

In Corollary (ref), we follow two core setups similar to the ones employed in 2022_Edelman_et_al. First, the complexity of the function class $\mathcal{F}$ is governed by the combined capacity of the Transformer blocks and the MLP output layer, and we use norm-based Transformer blocks with sparse MLP, where the $\ell_{2,1}$ norms of the Transformer weights are explicitly bounded. This enables the model complexity to scale only logarithmically in \(d\) and \(n\), both of which may be large in practice (e.g., \(d = 71\) and \(n = 97\) in our application of Section (ref)), as captured by the bound

where $d_m, D$ are the network width and depth of MLP that possibly grow with $T$, e.g., in Corollary (ref) we choose $D \lesssim \log T$. Second, we assume that the target function has a low-dimensional sparse structure: it depends only on an unknown subset of $s$ relevant input coordinates with $s$ being a finite number, and is $\beta$-H\"older smooth. Together, these assumptions imply that self-attention can identify and aggregate the relevant information from the high-dimensional input, while the MLP only needs to learn a smooth mapping on an $s$-dimensional signal. Hence, the following bias rate is attainable under Assumptions (ref) and (ref) for a given class $\mathcal{F}$ satisfying the aforementioned complexity bound:

Based on the bias-variance decomposition provided in Theorem (ref) and results outlined above, we can achieve the following convergence rate.

Empirical Study

Data

We study eight (see Table (ref)) U.S. global systemically important banks (G-SIBs), as identified by the Financial Stability Board (FSB) with the Basel Committee on Banking Supervision (BCBS) and national regulators. These banks are considered as “too big to fail" because of their role in the global financial system. Our sample covers daily log returns from October 2006 to November 2013, giving $T = 1776$ observations. The period includes major stress events or market drawdowns such as the 2008 financial crisis, the Goldman Sachs SEC lawsuit in 2010, the Greek debt crisis, and the U.S. debt ceiling disputes of 2011 and 2012. The sample period ensures us the diversity in news topics to study a set of systemic risks.

We use a set of common macro-state variables as risk factors for VaR estimation in (ref) 2016_Adrian_and_Brunnermeier, 2016_Haerdle_et_al: the Implied Volatility Index (VIX), daily returns on the S&P 500, the spread between Moody’s Baa Corporate Bond Yield and the 10-Year Treasury yield, and the yield spread between the 10-Year and 3-Month Treasury yields\footnote{VIX, S&P 500 returns, and stock prices are from Yahoo Finance. Credit spreads and yield spreads are from the Federal Reserve Bank of St. Louis (FRED).}. For CoVaR estimation, we use both bank stock returns with the financial news articles released from Reuters. The news dataset contains 105,359 articles, originally collected by 2014_Ding_et_al. After removing duplicates, we retain 97,264 articles.

On average, 38 articles are published per day, with daily counts ranging from 1 to 97 and a median of 39. Figure (ref) shows that article volume rises sharply during market stress and drawdowns, such as the 2008 crisis and the 2011--2012 U.S. debt ceiling standoff. These spikes coincide with declines in the S&P 500, suggesting that media coverage intensifies in volatile periods. The average article length is 449 words, with a median of 415.

Data Preprocessing

{Preparing the input for our model proceeds in three steps. First, we employ the pre-trained embedding model gemini-embedding-001 as the embedding mapping $\pi_{e}$ to convert $n$ ($n=97$) textual news used at time $t$ into numerical news embeddings $E_{j,t} \in \mathbb{R}^{d_{e} \times n}$ (we choose $d_{e}=64$). {For days on which $n$ is smaller than 97, we apply padding to the news embeddings (see the explanation of padding in the footnote in Section (ref)).} We consider all news articles observed up to time $t$ within a five-day look-back window. This choice is motivated by 2013_Garcia, who documents that market reactions to news typically reverse within four trading days, suggesting that such an effect is short-lived and largely driven by non-fundamental factors. Second, because the embeddings do not inherently have temporal order, we inject positional information to preserve the chronological information of the news sequence. Following 2017_Vaswani_et_al, we add fixed sinusoidal positional encodings (PE)\footnote{The positional encoding for position $k$ and dimension $i$ is defined as: $$

$$ where $k$ is the temporal position within the $5$-day look-back window. These encodings are scaled by the norm of the input embeddings.} to the sequence $E_{j,t}$. This preserves the temporal sequence information of news articles. Third, we augment the news embeddings with numerical return information by concatenating the returns of the remaining $J-1$ ($J = 8$) banks with $E_{j,t}$, yielding the return-augmented news embeddings $Z_{j,t} = \Pi(R_{-j,t}, E_{j,t})$. We then take $Z_{j,t}$ as our model input, which consists of 97 input tokens, each with 71 (i.e., 64 plus 7) dimensions.}

As mentioned above, we use gemini-embedding-001, a pre-trained embedding model released in June 2025 by GoogleAI. This model generates dense vectors of 3072 dimensions. It takes a maximum of 2048 tokens per input, which corresponds to about 1228--1638 English words\footnote{According to GoogleAI documentation, 100 tokens are about 60–80 English words.}. The majority of articles in our dataset are shorter than this limit, with only about 1% exceeding it (see Figure (ref)). The longest article has 9471 words.

Pre-trained embeddings are known to be effective general-purpose semantic representations 2025_Zhao_et_al and have shown good performance in various natural language processing tasks such as semantic search, text classification, and clustering 2025_Lee_et_al. An important property of gemini-embedding-001 is that the embedding dimensions are ordered by importance, with the most informative content appearing in the first dimensions. This property enables dimensionality reduction by keeping only the top leading dimensions of each vector embedding 2024_Kusupati_et_al, 2024_Kim_et_al. Following this idea, we keep the top 64 dimensions of each embedding, since using all dimensions would make the model too large and require much more training data. We can then represent a collection of $n$ articles by a semantic embedding matrix with size $\mathbb{R}^{d_{e} \times n}$, where each row is a $d_{e}=64$ dimensional embedding. This matrix is then used as the input in the downstream risk prediction task.

To validate that the embeddings capture meaningful semantic patterns in the text data, we study their spatial structure in the embedding space. Since the embedding model is designed to map semantically similar texts to nearby points in the vector space, we expect that texts with similar content will produce embeddings with smaller distances (e.g. Euclidean, cosine distances etc.). For example, the sentences “The market is going to crash" and “The market is expected to plunge" should yield embedding vectors that are close to each other, reflecting their semantic similarity.

First, we apply the t-distributed Stochastic Neighbor Embedding (t-SNE) algorithm to project the high-dimensional embeddings into two dimensions for visualization purposes. t-SNE is one of the most popular methods for revealing intrinsic structures in high-dimensional datasets, including trends and patterns, through a nonlinear dimensionality reduction technique 2022_Cai_and_Ma. This is useful for high-dimensional data that lie on several different but related low-dimensional manifolds 2008_Van_der_Maaten_and_Hinton. For example, news items can report various sub-events or highlight different facets within the same topic.

t-SNE includes a parameter called “perplexity”, which balances the focus between local and global structures among data points. Local structure refers to the relationships between each point and its nearest neighbors, capturing fine-grained similarities. Global structure refers to the broader arrangement of clusters relative to one another. Lower perplexity values emphasize local relationships, producing smaller, tighter clusters, whereas higher perplexity values emphasize global structure, resulting in larger, more diffuse clusters. As shown in Figure (ref), clusters become smaller and more concentrated with lower perplexity values, and larger and more diffuse when the algorithm emphasizes global structure. However, in all cases, the visualizations reveal a pattern of clustering, suggesting that the embeddings form coherent groups in the reduced space.

Next, we apply the KMeans clustering algorithm directly in the original high-dimensional embedding space. Since KMeans is an unsupervised learning algorithm, the number of clusters must be specified in advance. Choosing a smaller number of clusters results in broader, more general themes, while a larger number yields more granular clusters with narrower topics. As an example, we examine all 6124 news articles related to Goldman Sachs\footnote{These articles are selected based on whether the title, summary, or body text contains the phrase “Goldman Sachs.”}. Figure (ref) shows the result of applying KMeans with 30 clusters to these articles. For each cluster, we use the LLM gemini-2.0-flash to assign a common theme to the 15 most representative articles. These representative articles are defined as those closest to the cluster centroid. The LLM generates a thematic summary based on the topic shared among the 15 selected articles.

Taken together, the t-SNE visualizations and the KMeans clustering results provide qualitative evidence that the embeddings capture meaningful semantic structure in the news corpus. The presence of stable clustering patterns across different perplexity settings, along with the emergence of interpretable themes within clusters in the original embedding space, suggests that semantically related articles are mapped to nearby regions.

CoVaR Prediction

We input the high-dimensional return-augmented news embeddings into our Transformer-based model with one Transformer-block layer and a two-layer MLP. We set the hidden dimensions of both the FFN and the MLP to $d_h = d_m = 64$. We estimate CoVaR in two steps as described in Section (ref). First, we estimate $\operatorname{VaR}$ for all banks using linear quantile regression on macro-state variables, following 2016_Adrian_and_Brunnermeier and 2016_Haerdle_et_al. Second, we estimate $\operatorname{CoVaR}$ for our target bank using a Transformer-based model that incorporates both numerical (i.e., stock returns) and textual information (i.e., financial news articles).

Our primary objective is to evaluate the added value of incorporating textual information in the second step of CoVaR framework. The dependent variable is the log of the stock return of the target bank. The independent variables include the contemporaneous log stock returns of other banks and news embeddings from the past five days.

To evaluate out-of-sample performance, we split the dataset into three subsets: training ($\mathcal{D}_{1}$), validation ($\mathcal{D}_{2}$), and test ($\mathcal{D}_{3}$). The training set covers 40% of the full dataset and is used to fit the model. The validation set takes 20% and is used for regularization (i.e., early stopping) and hyperparameter tuning. We use the remaining 40% as the test set to evaluate out-of-sample predictive performance of our Transformer. Each subset contains at least one major stock market downturn. We highlight the first and third big market downturns with gray-shaded areas in the following figures and skip the second one, since the second occurs in the validation set and is not of interest to us. The first downturn corresponds to the 2008 financial crisis, which is determined by the NBER (National Bureau of Economic Research) recession period. The third downturn corresponds to the 2011 U.S. debt ceiling crisis, which we determine using quarterly windows. Note that we train the model only once and then use the trained model to perform one-step-ahead predictions on the test data in a rolling-window manner. This means that for each test period, we use all available information up to time $t$ to predict the outcome at time $t+1$.

We train our Transformer-based model using SGD with mini-batches 2025_Marek_et_al. In the hyperparameter search, we experiment with three learning rates (0.00015, 0.0015, and 0.015) and three batch sizes (32, 64, 128). We then select the optimal hyperparameters that achieve the lowest loss on the validation data. We choose 200 training epochs with early stopping when the loss does not decrease for 50 consecutive epochs. Because our Transformer incorporates textual information, the optimization becomes much more difficult, and the training objective may include multiple local minima. We acknowledge that some training runs may not converge to the same solution.

Figure (ref) shows the $\operatorname{CoVaR}$ and $\operatorname{\Delta CoVaR}$ estimates for Goldman Sachs, with and without textual information (see Figure (ref) in Appendix (ref) for other assets). {$\operatorname{\Delta CoVaR}$ is an alternative risk measure proposed by 2016_Adrian_and_Brunnermeier that builds upon the CoVaR framework. It quantifies the spillover effect by measuring the change in the system's risk (CoVaR) when a specific bank $j$ shifts from its median state to a state of financial distress. We denote $\operatorname{\Delta CoVaR}$ as

Importantly in this context, the $\tau$ used for the conditional systemic risk (CoVaR) and the $\tau$ defining the banks' distress levels (VaR) can be distinct.}

The text-based CoVaR is estimated using our Transformer-based model introduced in Section (ref), and the non-text-based (i.e., return-only) estimates are computed from a two-layer MLP neural network with hidden dimensions 64. During periods of non-crisis and normal market activity, the gap between VaR and CoVaR tends to widen, reflecting the silent accumulation of systemic risk. This supports the perspective of 2016_Adrian_and_Brunnermeier, who argue that systemic risk accumulates silently during calm periods and materializes during crises. The latter is evident during market downturns, where the gap narrows as individual institutions approach their VaR thresholds, reflecting the realization of systemic risk.

Comparing the Transformer-based CoVaR model with text against the baseline MLP without text shows that incorporating financial news leads to noticeably different risk estimates during periods of market stress (See Figure (ref)). The largest absolute differences appear during the 2008 crisis, especially in October, when the differences reach about 0.06--0.065 for both CoVaR and $\Delta$CoVaR. At that time, the global markets were highly unstable and concern about the viability of major financial institutions intensified.

During the 2008 crisis, the media narrative was dominated by panic about the global banking system, fears of a deep recession, and widespread uncertainty about whether government interventions would be sufficient to stabilize financial markets. For example, Reuters reported news such as “Germany slashes 2009 growth forecast as crisis bites", “Recession looms despite global interventions”, and “Swiss banks raise emergency funds to fight crisis”. These news articles carry signals about systemic fragility, which push the model toward more negative CoVaR and $\Delta$CoVaR estimates compared with the model that relies purely on numerical variables. Other late-2008 dates (e.g., November, December) show similar gaps. In relative terms, the two models differ by 40--50% during this crisis.

Another significant drop shows up in mid-2010. The news then focused on the euro-zone debt crisis and systemic risk in Europe. For example: “EU urges Greece to stick to austerity, pension plan”, “Euro-zone troubles may roil stocks”, and reports on flash-trading risks such as “Naked swaps, flash trading and other key terms.” This information again increase perceived risk, and the text-based model reflects that in more negative CoVaR estimates.

For the out-of-sample period, the largest absolute differences occur during the volatility episode from July 2011 to January 2012. The absolute gaps reach about 0.033--0.037 for CoVaR and $\Delta$CoVaR, with relative deviations of around 33% to 40% for $\Delta$CoVaR. The dominant news themes during this time concern the U.S. debt-ceiling crisis, such as “United States loses prized AAA credit rating from S&P” and “Obama officials attack S&P’s credibility after downgrade”, as well as the euro-zone sovereign crisis, such as “Euro zone investors gloomiest for nearly 2 years” and “ICE bars MF Global from floor, customers can unwind only”\footnote{Based on Reuters news reports, the collapse of MF Global in 2011 was caused by risky 6 billion euro bets on European sovereign bonds, heavy leverage, and the alleged misuse of client funds to cover losses, culminating in bankruptcy.}.

Across both crisis periods (and also the one in 2010 included in the validation set), the sign of the differences is consistently negative, showing that the model with text produces more negative CoVaR and $\Delta$CoVaR estimates than the model without text. In other words, during crisis episodes, incorporating financial news systematically amplifies the perceived downside tail risk, particularly during crisis episodes. We show an extra news content analysis in Appendix (ref) to better understand why the inclusion of news embeddings leads to more negative CoVaR predictions during periods of financial distress.

Average Quantile Loss

We show that our CoVaR measure can be accurately estimated using a residual-based analysis. In particular, we evaluate the average quantile (AVQ) loss on the test data for CoVaR models with and without textual information. Starting from January 21, 2011 (the beginning of the test period), we compute quarterly average losses and extend the evaluation cumulatively every three months.

Table (ref) reports the loss comparison results for Goldman Sachs. Over the full test period, the text-based CoVaR model exhibits slightly lower losses than the CoVaR model without text. However, we observe a noticeable increase in losses between the 9th and 15th months for both models. These periods cover exactly a major market downturn: on August 5, 2011, Standard & Poor’s downgraded U.S. sovereign debt from AAA to AA+ for the first time in history; and later, until December 2011, the U.S. debt limit was raised three times after Congress failed to block the increases or pass a Balanced Budget Amendment\footnote{See details in Congressional Research Service \href{https://www.congress.gov/crs_external_products/RL/PDF/RL31967/RL31967.139.pdf}{The Debt Limit: History and Recent Increases}, and U.S. Government Accountability Office \href{https://www.gao.gov/products/gao-12-701}{Debt Limit: Analysis of 2011–2012 Actions}}.

Reuters news articles during this period documented the intense political conflict between Congress and the Obama administration over raising the debt ceiling, raising fears of a potential default. The textual data from this time contains information relevant to systemic stress: terms such as “default risk,” “debt ceiling,” and “downgrade” become frequent in financial news. The Transformer-based model takes this textual information as input and achieves a smaller increase in loss during this crisis period.

We report the AVQ losses for all the banks in Table (ref). Overall, the two models show broadly similar total losses on test data. The inclusion of the financial news does not dramatically change overall predictive accuracy across the full test sample. The performance differences between the two models are generally small across the eight assets. However, we observe two things: 1) the differences between the text-based CoVaR and the CoVaR without text consistently display a negative dip for all banks during the crisis, regardless of whether the total test losses are higher or lower for either model (See Figure (ref)); and 2) most losses in the first 12 months, which include the major market downturn, tend to be lower or at least comparable for the Transformer-based model with text.

The generally similar total losses suggest that news content may be noisier and provide limited incremental information for risk prediction during stable periods, which may offset the benefits of text integration in evaluations. However, when we focus on the first 12 months, which cover the crisis period, the lower losses observed during this time indicate that textual information becomes more valuable in more uncertain market environments.

Comparison with Sentiment-Based Measures

We compare our results with sentiment-based CoVaR estimates. Specifically, we use FinBERT, a financial-domain pre-trained language model, to classify each news article into one of three sentiment categories: positive, negative, or neutral. These sentiment information are then incorporated into two alternative modeling frameworks: MLP and Transformer. We estimate CoVaR using each sentiment-based model and compare the resulting MLP- and Transformer-based sentiment CoVaR estimates with our text-based CoVaR estimates.

For the first method, to let the MLP take sentiment as an input variable, we construct the daily sentiment index following a commonly used method in the literature 2007_Tetlock, 2011_Loughran_and_McDonald, 2014_Nassirtoussi_et_al:

where “\#" denotes the number of articles in each sentiment category on day $t$. The index reflects the net sentiment (positive minus negative) scaled by the total number of articles, representing the overall sentiment of financial news on that day (see the top panel of Figure (ref)). The observed sentiment index fluctuates around zero and lies within the range $[-0.5, +0.5]$. Negative sentiment occurs during crisis periods, but the magnitude is small. We augment the return-only MLP model by adding the sentiment index as an additional input variable.

As shown in the middle panel of Figure (ref), the sentiment-based CoVaR adds limited value to systemic risk estimation compared to the return-only MLP CoVaR model. We provide some potential explanations. First, sentiment is only one aspect of human language and cannot fully capture the complexity of language. Relying only on sentiment leads to the loss of important information relevant to financial markets. Second, even if we assume sentiment is the only key information in textual data, investor sentiment still varies significantly across contexts and is not uniformly informative 2024_Wang_and_Ma. Third, common aggregation methods such as averaging or polarity counts are typically static and equal-weighted. They treat all news sentiments as equally important through the time. This overlooks time-varying importance in news information. These aggregation processes lead to the homogenization of sentiment index 2024_Wang_and_Ma. This homogenization can be both cross-sectional and temporal: (1) The sentiment scores tend to converge toward the mean and thereby smoothing out the distinctive crucial information in individual news items. The extreme sentiments are diluted, causing predictive information loss; (2) The similarity in average scores across different days homogenizes daily news sentiments, making it difficult for models to capture meaningful time-series variation in investor sentiment. The homogenization mechanism helps explain the small magnitude of day-to-day sentiment fluctuations shown in the top panel of Figure (ref). Moreover, news and prices may have interactions in real world, which the common sentiment-based approaches fail to capture.

The second method we use computes a sentiment-based Transformer CoVaR. Instead of feeding high-dimensional text embeddings, we map each document simply as one of three numerical sentiment labels (e.g., 1, 2, 3) to represent negative, neutral, and positive. Then we apply the same procedure as the text-based Transformer to construct the input matrix.

We obtain the same conclusion as using the first sentiment-based MLP model: the sentiment-based CoVaR adds limited value to systemic risk estimation compared to the return-only MLP CoVaR model. The largest differences between the two models occur during the 2008 financial crisis. We zoom in and check the sentiment classifications of the news. We zoom in and check the sentiment classifications of the news. Indeed, some news are classified as negative when they are extremely negative, such as “AIG struggles to survive financial tsunami”. However, there are also news classified as positive, such as “AIG gets New York's help in accessing \$20 billion”. But is this really a positive news? One can argue that AIG receives governmental support, but on the other hand, when an institution really needs government intervention, it also means the situation is already extremely bad. In addition, “Chrysler sees U.S. auto market gains toward end 2009” receives a positive label despite the first half of the article describing severe declines in U.S. auto sales, such as “… auto sales were down more than 11 percent through the first eight months …”. Classifying articles into only three categories will ignore the information in the first part. Moreover, more than 50% news during crisis are simply classified as neutral. Thus, collapsing articles into three sentiment categories discards the nuances contained in mixed-tone narratives.

The limitations of the sentiment-based CoVaRs become obvious, when comparing them with the text-based Transformer CoVaR estimates (i.e. the one that uses text embedding). The text embeddings with richer representations that capture deeper linguistic and contextual information, allowing them to detect more relevant information in news, especially during periods of market stress. As shown in the bottom panels of Figures (ref) and (ref), the differences between the sentiment-based models and the text-based Transformer are most obvious during crisis periods, where negative dips in CoVaR estimates reappear. This is similar to the differences observed between text-based and return-only CoVaR estimates. This shows that sentiment-based CoVaR modeling overlook important information present in the text, which on the contrary can be captured by using text embeddings.

Look-Ahead Bias

Applying the GoogleAI embedding model to transform news articles into embedding vectors may introduce a look-ahead bias. The gemini-embedding-001 model may have been trained on data that include future information beyond the analysis period. For instance, if the training data covers the 2008 financial crisis and subsequent bank failures, the model may produce highly similar embeddings for news about different banks. This can be problematic if it occurs even for articles published before the crisis, when news mentioned banks were operating normally. Such bias may skew the embeddings, which overlook subtle but important distinctions in news across different financial institutions.

Mitigating the look-ahead bias is challenging. Ideally, we would use a language model trained only on data preceding our evaluation period. However, since the LLM boom began after 2020, it is difficult to find a model that meets the data requirements of our application and contains sufficient coverage of crisis periods for analysis. Developing a new LLM would be computationally and resource intensive. In addition, GoogleAI does not disclose the knowledge cutoff date of gemini-embedding-001. One dataset used in its training is FRet, which includes diverse topics such as blog posts, news articles, Wikipedia-like content, and forum discussions 2024_Lee_et_al. Therefore, there is a high potential for look-ahead bias.

We address this bias by masking the names of the eight banks and other identifying information to prevent the model from exploiting bank-specific cues. Similar to 2025_Breitung_and_Mueller and 2023_Glasserman_and_Lin, we apply a Named Entity Recognition (NER) model from the Python package spaCy to detect the identifying information of banks. 2025_Breitung_and_Mueller empirically demonstrate that using this advanced model ensures high masking accuracy.

We find that masking or not masking identifying information does not change the overall conclusion: the CoVaR still becomes more negative during crisis periods when financial news is incorporated. However, there are small numerical differences remain between the models with and without masking (see Figure (ref)). Negative values in the figure indicate that the model without masking produces more negative CoVaR estimates. The largest absolute differences occur between October and November 2008, when the CoVaR and $\Delta$CoVaR gaps reach about 0.022--0.027. In relative terms, the Transformer-based model without using masking deviates from the Transformer-based model using masking by roughly 12--14% for both CoVaR and $\Delta$CoVaR. Over the out-of-sample period, the largest gap appears from August to October 2011, with absolute differences of about 0.015 for CoVaR and $\Delta$CoVaR (roughly 9--12% in relative terms). A likely explanation is that masking reduces how often multiple banks appear in the same article, which weakens the model’s ability to infer relationships between institutions. This may affect the systemic risk estimates.

Conclusion

We develop a Transformer-based framework for systemic risk modeling that integrates both structured financial data and unstructured textual data. Our approach extends the CoVaR methodology to incorporate high-dimensional, multimodal inputs. We provide empirical evidence of improved tail-risk estimation during periods of financial stress. Specifically, we show that including textual data from financial news results in more negative CoVaR and $\Delta$CoVaR values during crises under a better predictive performance, highlighting the importance of market-related information contained in textual data. In addition, forward-CoVaR, as another systematic risk measure used for early warning, can be interesting for future empirical work.

We establish an upper bound on the $\ell_2$ risk of the Transformer-based quantile regression. In particular, we also provide the convergence rate of our procedure for an explicit case, demonstrating that it can efficiently learn complex nonlinear dependencies in high-dimensional settings. This complements existing literature on deep learning for financial risk modeling and provides a rigorous econometric foundation for integrating Transformers into systemic risk analysis.