Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
172,833 characters · 17 sections · 70 citation commands
A Nonlinear Target-Factor Model with Attention Mechanism for Mixed-Frequency Data
\thispagestyle{empty}
Keywords: Mixed-frequency, Factor models, Transformers, Attention,\\ Macroeconomic forecasting, Nonlinearity
Panel datasets with large cross-sectional and temporal dimensions are common in macroeconomics, finance, and other social sciences. Such datasets are often well characterized by an approximate factor structure, in which a few latent factors capture most of the comovement across units. Traditionally, these latent factors are estimated directly from the panel of primary interest, referred to as the target data. However, in modern empirical settings, there are often additional or auxiliary panels that share relevant information or common factors with the target panel. Integrating information across such panels can improve the quality of factor estimation for the target data and improve downstream tasks such as forecasting. Moreover, auxiliary panels can help identify factors that influence only a limited subset of units or that are otherwise undetectable due to sparsity or missing data in the target sample. This idea of leveraging auxiliary datasets to enhance inference for a specific target dataset is closely related to the concept of transfer learning, which has proven effective in modern machine learning (weiss2016survey).
Sharing relevant information across target and auxiliary datasets also arises in a wide range of applied settings. A prominent example is mixed-frequency data, which we examine empirically in this paper. High-frequency information can be exploited to improve inference for lower-frequency target variables (forni2001generalized, ghysels2007midas). Related information-sharing structures arise naturally in firm-level and macro-financial panels, where cross-sectional and aggregate data jointly inform economic dynamics (stock2002forecasting, kelly2019characteristics). Similar challenges arise in energy and climate applications, which involve large, heterogeneous panels observed at different temporal and spatial resolutions (grassi2017key).
To illustrate, many macroeconomic variables, such as Gross Domestic Product (GDP), are observed only at quarterly or lower frequencies, whereas others are available at higher frequencies. When some of the factors driving higher-frequency indicators are correlated with macroeconomic dynamics, information from the high-frequency data can be exploited to infer higher-frequency factors and forecast future values of the target series in the target panel.
However, classical factor models and principal component analysis (PCA) techniques are typically designed under the assumptions of homogeneous sampling frequencies and linear signal extraction. Such assumptions limit their applicability to mixed-frequency settings, where data are observed at uneven temporal resolutions across variables. In practice, high-frequency information is often collapsed into coarse aggregates, such as monthly or quarterly averages, prior to estimating the factor model. While convenient, this aggregation procedure inevitably discards valuable cross-sectional and temporal heterogeneity in the original high-frequency data. As a result, the estimated factors may fail to capture short-term dynamics, asynchronous adjustments, or heterogeneous responses across units, leading to biased or inefficient inference in mixed-frequency settings.
In this paper, we propose the Mixed-Panels-Transformer Encoder (MPTE), which overcomes the aforementioned limitations of classical factor models. We summarize our contributions as follows. First, we use attention mechanisms to enable context-aware signal construction through flexible, input-dependent weighting schemes. Second, we embed these attention mechanisms within a Transformer Encoder architecture, allowing the model to capture nonlinear signals through stacked attention and feedforward layers\footnote{By “feedforward layer”, we refer to the position-wise fully connected subnetwork commonly used in Transformer models, also known as a multilayer perceptron (MLP)}. Third, for the case of linear activation functions, we develop an inferential theory and establish consistency, efficiency, and asymptotic normality for factor and loading estimators based on attended inputs.
A growing body of econometric and machine-learning literature applies selective weighting across the cross-section that is conceptually analogous to modern attention mechanisms. In classical factor models, Projected PCA fan2016projected introduces a kernel matrix that reweighs units according to observable characteristics, effectively implementing a similarity-based cross-sectional smoothing that is conceptually analogous to attention mechanisms. pelger2024target_pca propose Target PCA, an extension of standard PCA that jointly estimates signals for target and auxiliary panels. This framework provides direct motivation for our approach. They apply PCA to the weighted average of the second-moment matrices of the two panels, where the optimal weighting coefficient is a scalar chosen to ensure consistent and efficient estimation of the target factors. connor2012efficient rely on kernel weight matrices that assign higher weights to units sharing similar covariates, a mechanism conceptually related to attention scores. Related approaches include weighted and sparse PCA methods zou2006sparse, which modify the eigen-decomposition by imposing sparsity and correspond to hard cross-sectional attention. kelly2021autoencoder propose an autoencoder-based stochastic discount factor estimator that learns feature-specific scaling functions, which serve as data-dependent attention weights across assets.
In mixed-frequency and Mixed-Data Sampling (MIDAS) regressions (ghysels2007midas, foroni2015unrestricted) high-frequency observations enter through parametrized lag weights that selectively emphasize the most informative time periods, closely resembling temporal attention scores. Distributed lag and kernel-smoothing estimators nadaraya1964estimating, fan1996study apply localized temporal weights, effectively focusing estimation on nearby or relevant points, in a manner conceptually analogous to attention over sequences. stock2016dynamic provide an extensive review of the literature on dynamic factor models, in which information is reweighted over time through stochastic volatility, state-dependent dynamics, or regime switching, which can be interpreted as adaptive temporal attention.
There are also a limited number of papers that weigh both the cross-sectional and time dimensions. wang2019factor apply kernel-weighting to cross-sectional and time dimensions.bai2015identification use time-varying loadings and cross-sectional smoothing. chen2022factor employ separate weights along each mode in tensor factor models, which is conceptually analogous to multi-head attention across dimensions.
More broadly, recent work interprets Transformer attention as a normalized kernel smoother tsai-etal-2019-transformer,katharopoulos2020transformers, strengthening the conceptual link between attention mechanisms and econometric approaches that selectively weigh cross-sectional information.
Building on these selective weighting ideas, we propose a factor framework with general cross-sectional and temporal attention matrices that explicitly accommodates rich heterogeneity in both the target and auxiliary panels. In many empirical settings, auxiliary data are informative for the target task only selectively, for example, when only subsets of units, periods, or factor directions in an auxiliary panel carry signals about the target outcomes, while others contribute primarily noise. Such heterogeneity may arise from differences in economic relevance across units, time-varying predictive content, partial alignment between auxiliary and target factor spaces, or structured heteroskedasticity in idiosyncratic errors. Standard block-based approaches, including Target PCA by pelger2024target_pca, implicitly treat units and periods symmetrically within each block and therefore cannot exploit these forms of selective informativeness without prior dimension reduction or ad hoc preprocessing. By contrast, our attention-based formulation allows the data to determine which cross-sectional units and time periods are most informative for the target task, yielding an effective sample size that reflects relevance rather than raw dimension. This flexibility enables first-order efficiency gains from auxiliary information in genuine transfer-learning regimes, while still preserving a clear notion of target-specific factors identified by the target block. As a result, our framework strictly generalizes existing target-focused PCA methods and is particularly well suited for applications with heterogeneous relevance, structured noise, and partial factor overlap across datasets.
An important distinction between our framework and existing target-focused PCA methods is its adaptivity to heterogeneous relevance across units and time. Conceptually, our approach is analogous to adaptive thresholding and shrinkage procedures in high-dimensional statistics, which improve efficiency by learning which components carry signal rather than imposing fixed selection rules. While pelger2024target_pca rely on block-level structure and uniform weighting to ensure robustness when the target panel may be partially observed, they cannot adjust to varying degrees of informativeness within or across blocks. By contrast, the attention matrices in our framework allow the estimator to adaptively reweight cross-sectional units and time periods based on their relevance for the target task.
Our work contributes to three different streams of literature. First, we contribute to the literature on large-dimensional factor models. Our paper provides a context-aware nonlinear generalization of pelger2024target_pca. We allow nonlinear signals and enable the model to learn the context by modifying the inputs through cross-sectional and temporal weighting matrices. The former allows the model to learn the context by using contemporaneous information about all series, while the latter helps the model learn temporal context by using information from all past embeddings. The machine learning literature has long acknowledged the close relationship between autoencoders and principal component analysis (PCA) (e.g., baldi1989neural), and by extension their connection to latent factor models in asset pricing kelly2021autoencoder. In particular, a linear autoencoder with a hidden layer of $K$ neurons is a linear factor model with $K$ latent factors. Our paper uses this connection to build a theory for the case of linear activation functions, to allow direct comparison with the methodology of pelger2024target_pca, while keeping the flexibility of nonlinear signals for simulations and empirical applications.
Second, we contribute to the literature on nonparametric techniques (such as kernel regression, nadaraya1964estimating) and attention mechanisms. In traditional factor models and regression models, each input contributes to the output through fixed weights. Attention mechanisms replace these with data-dependent weights that adapt to context. The model “attends” more to informative observations and less to irrelevant ones. In this context, attention acts as a learned smoothing kernel that reweighs observations based on similarity. In contrast to a traditional kernel, it is fully data-driven and is less affected by the curse of dimensionality. In fact, this interpretation extends beyond factor models. Recent work has established connections between classical regression methods, such as ordinary least squares (OLS), and attention mechanisms. coulombe2025ordinary show that OLS can be reinterpreted as a special case of an attention mechanism. Specifically, when OLS is viewed as selecting similarities in a transformed regressor space, rather than as purely estimating coefficients via partial correlations, the resulting solution is analogous to a query--key--value structure in attention. In this paper, we establish a connection between traditional factor models estimated using PCA and attention mechanisms. To the best of our knowledge, this is the first work to study the theoretical properties of PCA under “attended” inputs.
Finally, we contribute to the literature on mixed-frequency data. Our paper can be interpreted as a nonlinear and context-aware generalization of MIDAS methods. When the predictors, $X_t$, and targets, $Y_t$, are sampled at different frequencies, traditional approaches typically require manual alignment, while MIDAS specifies a parametric weighting structure over the high-frequency lags. Our framework embeds both $X_t$ and $Y_t$ in a unified sequence for the Transformer encoder architecture. The model learns how to aggregate information across frequencies via attention, without resampling or pre-specified weight schemes. A small number of applied papers augment the MIDAS framework with nonlinearities. lin2024ijf propose a neural network approach that handles mixed-frequency data through a shared encoder and dual decoders; dai2024freqformer introduce FreqFormer, a frequency-aware Transformer developed for image super-resolution; ji2025novel develop QRTransformer--MIDAS, which integrates quantile regression into a Transformer with MIDAS-based high-frequency preprocessing. In contrast, our approach adopts an encoder-only Transformer architecture, is designed explicitly for mixed-frequency time-series panels, and learns cross-frequency aggregation directly through attention, without relying on MIDAS-style preprocessing or decoder-based generative structures.
Our framework builds on recent advances in Transformer-based models and offers a novel approach to estimating factor models with mixed-frequency data. We shift the focus from averaging information, as commonly done in classical factor models, to attending to information. We also provide an extensive ablation study that examines the role of each component of MPTE in both simulations and empirical applications. Specifically, we study the role of attention mechanisms, nonlinearities, the use of high-frequency data for forecasting low-frequency targets, and additional architectural components such as temporal encoding.
To the best of our knowledge, this is the first paper to disentangle and interpret the cross-sectional and temporal aggregation patterns learned by a transformer in macroeconomic forecasting, and to link them to economically interpretable sources of predictive power. By averaging attention weights across variables and across time, we recover target-specific measures of variable importance and lag relevance that vary systematically with model architecture and forecast target. We show that nonlinear transformations and explicit temporal encoding allow the model to place disciplined weight on state-dependent indicators and medium- to long-horizon dynamics, while simpler specifications rely predominantly on short-run fluctuations and diffuse cross-sectional signals. These results suggest that attention-based models can do more than improve forecast accuracy: they provide a transparent, data-driven framework for assessing which economic variables and horizons are most relevant for forecasting particular macroeconomic outcomes, with direct implications for policy monitoring and analysis.
The rest of the paper is structured as follows. Section (ref) introduces the MPTE framework and formalizes the attention-based aggregation of cross-sectional and temporal information. Section (ref) develops the inferential theory for the linear version of the model and establishes consistency and asymptotic properties of the resulting estimators. Section (ref) extends the framework to nonlinear signal structures. Section (ref) describes the Transformer encoder implementation of MPTE and discusses practical considerations for handling nonlinearities and mixed frequencies. Section (ref) presents simulation evidence assessing the performance of attention-based mixed-frequency estimation under linear and nonlinear signal structures. Section (ref) provides an empirical application to U.S.\ macroeconomic data. Section (ref) concludes.
This section presents MPTE and its associated estimator, which integrates target and auxiliary datasets through cross-sectional and temporal attention operators. We discuss the connection to Target PCA and show in Section (ref) that Target PCA arises as a special case of a Transformer encoder.
The target panel dataset Y has $T_y$ time periods and $N_y$ cross-sectional units. $Y \in \mathbb{R}^{T_y\times N_y}$ has the following signal-plus-noise structure:
where $Y_{tj}$ denotes the data for the $j$-th cross-sectional unit at time $t$, $(\Psi_{y})_{tj}$ and $(e_y)_{tj}$ are the common and idiosyncratic components of $Y_{tj}$, respectively. Equation (ref) can be written in a matrix notation:
In addition, an auxiliary panel dataset X is available that has $T_x$ time periods and $N_x$ cross-sectional units. $X \in \mathbb{R}^{T_x\times N_x}$ has the following signal-plus-noise structure:
where $X_{ti}$ denotes the data for the $i$-th cross-sectional unit at time $t$, $(\Psi_{x})_{ti}$ and $(e_x)_{ti}$ are the common and idiosyncratic components of $X_{ti}$, respectively. Equation (ref) can be written in a matrix notation:
pelger2024target_pca use a linear factor model for signal components of $Y$ and $X$, with the number of latent factors denoted $k_y$ and $k_x$, respectively. Specifically, they assume $\Psi_y = F_y(\Lambda_{y}^{Y})^{\top}$ and $\Psi_x = F_x(\Lambda_{x}^{X})^{\top}$, where $F_y$ and $F_x$ are $T_y \times k_y$ and $T_x \times k_x$ matrices of latent factors, and $\Lambda_{y}^Y$ and $\Lambda_{x}^X$ are $N_y \times k_y$ and $N_x \times k_x$ matrices of factor loadings. Under linear signal structures and assuming $T_x=T_y=T$, equations (ref) and (ref) become:
They use the union of the factors in $Y$ and $X$, concatenated in a matrix $F \equiv [F_x, F_y] $, and apply PCA to the weighted average of the second moment matrices of both panels by solving the following optimization problem:
where $\Lambda_{y}$ and $\Lambda_{x}$ are $N_y \times k$ and $N_x \times k$ matrices of factor loadings. The weighting parameter $\gamma$ is selected to ensure consistent and efficient estimation of target factors. Let $Z^{\gamma}\equiv [ X \quad \sqrt{\gamma}Y ] $, then (ref) can be viewed as applying PCA to $(Z^{\gamma})^\top Z^{\gamma}$.
We augment the framework in (ref), (ref), and (ref) in three respects: (i) we use cross-sectional and temporal attention mechanisms to enable context-aware signal construction; (ii) we allow nonlinear signals $\Psi_y$ and $\Psi_x$; (iii) we relax the assumption $T_x=T_y=T$ required in pelger2024target_pca.
At the core of the Transformer architecture is the attention mechanism that was originally developed as a complement to recurrent neural networks (RNNs) for machine translation bahdanau2014neural. However, it was later shown to be effective as a standalone architecture even without the recurrent structure vaswani2017attention.
Let $Z\equiv [ X \quad Y ] $. The goal of attention is to map inputs $Z$ to a new set of inputs $\widetilde{Z}$ in a new embedding\footnote{An embedding refers to a learned vector representation of the data in which relevant structure is preserved for comparison, aggregation, or prediction. In attention-based models, embeddings provide the space in which similarity and relevance across observations are computed, without imposing a probabilistic or structural interpretation.} space that captures a richer structure. Enhancing the inputs helps extract better signals for target and auxiliary datasets in (ref) and (ref). For now, let $T_x=T_y=T$. We show in Section (ref) that this assumption is not necessary for implementing our approach; however, since pelger2024target_pca impose this restriction to solve (ref), we maintain it here to draw parallels with their method. Consider the simplest form of attention:
where $A \in \mathbb{R}^{T \times T}$. When $A=I_T$, where $I_T$ is a $T\times T$ identity matrix, this corresponds to the case without attention. When $A = ZZ^\top$, the resulting weights correspond to an unnormalized similarity matrix based on inner products, measuring pairwise similarity between observations.
Attention acts as a learned, data-driven smoothing kernel that reweighs information based on similarity. In contrast to traditional models, where inputs contribute through fixed weights, attention assigns data-dependent weights that adapt to context, allowing the model to attend more to informative observations and less to irrelevant ones.
vaswani2017attention introduce terminology from information retrieval to provide a more granular interpretation of (ref). In particular, inputs are linearly transformed into three objects: keys, values, and queries. Keys are defined as $K = Z W^k$ and summarize the attributes of each unit through a linear transformation of the inputs using transformation weights $W^k$. Values are defined as $V = Z W^v$ and represent the information associated with each unit that is aggregated through attention using weights $W^v$. Queries are defined as $Q = Z W^q$ and encode which attributes are relevant for forming the weighted combination of values using weights $W^q$. In the simplest case, the transformation matrices are identity matrices, so that $Q=K=V=Z$.
Using this terminology, the goal of attention is to measure the degree of match between queries and keys and use it to weigh the influence of the value matrix on the updated set of inputs $\widetilde{Z}$. We can rewrite (ref) as
where attention is computed as a row-wise normalized dot product between queries and keys, $A = \text{softmax}(QK^\top)$.
Our first contribution is to allow general temporal and cross-sectional attention matrices when learning signals in the target dataset. We consider the following transformation of the data matrix:
where $B \in \mathbb{R}^{T \times T}$ is a temporal attention matrix, and $A_z \in \mathbb{R}^{(N_x+N_y) \times (N_x+N_y)}$ is a cross-sectional attention matrix. The approach in pelger2024target_pca can be viewed as a special case with $B=I_T$ and $A_z = \text{diag}(I_{N_x}, \gamma I_{N_y})$. In contrast, we allow both $B$ and $A_z$ to be data-driven and, in the linear case, study the restrictions required to ensure consistent estimation of the signal components in (ref) and (ref).
The factor model for $\widetilde{Z} = B
A_z$ can be written as:
where $F^{(B)} := B F$ denotes the temporally transformed latent factor process, and $F_t^{(B)}$ denotes its $t$-th row. Define the concatenated idiosyncratic error matrix \[ e \equiv \bigl[e_x \ \ e_y\bigr] \in \mathbb{R}^{T\times (N_x+N_y)}. \] Given a cross-sectional attention matrix $A_z$, define the cross-sectionally transformed idiosyncratic component by \[ e^{(A)} \equiv e A_z \in \mathbb{R}^{T\times (N_x+N_y)}, \] and let $e_i^{(A)} \equiv (e^{(A)})_{:i}\in\mathbb{R}^{T}$ denote the $i$-th transformed idiosyncratic error series. With temporal attention $B\in\mathbb{R}^{T\times T}$, the fully transformed idiosyncratic component is \[ \widetilde e \equiv e^{(A,B)} \equiv B e^{(A)} = B[e_x\ \ e_y]A_z, \] so that $(B e_i^{(A)})_t = \widetilde e_{ti}$. Let $e_t := (e_{x,t}^\top,\, e_{y,t}^\top)^\top \in \mathbb{R}^{N_x+N_y}$ denote the stacked idiosyncratic vector at time $t$, where $e_{x,t}^\top$ and $e_{y,t}^\top$ are the $t$th rows of $e_x$ and $e_y$, respectively.
If $A_z$ is block diagonal, \[ A_z =
, \] we get the framework of pelger2024target_pca if we set $A_1 =
$ and $A_2 =
$. Then $\widetilde{Z}$ can be written as
We propose to apply PCA to the second moment matrix of the “attended” inputs $\widetilde{Z}$ in (ref). The PCA objective function is:
We call the aforementioned framework used to estimate the signals using (ref) MPTE. The encoding is achieved through PCA estimation of the signal under linear activations. The extension to nonlinear activations through Transformer architecture as well as theoretical equivalence between an autoencoder with $k$ hidden layers and a linear factor model with $k$ latent factors is studied in Section (ref).
Equation (ref) is equivalent to
With the identifying assumption $\Lambda^{(A)\top} \Lambda^{(A)}/(N_x+N_y) = I_k$, we can concentrate out $F^{(B)}$ and obtain the following objective function:
We assume $B$ preserves the factor space such that the spectrum of $k \times k$ matrix $T^{-1}F^\top B^\top BF$ is bounded away from zero and infinity. That is, there exist constants $0<c<C<\infty$ such that $c \leq \lambda_{\text{min}}(T^{-1}F^\top B^\top BF) \leq \lambda_{\text{max}}(T^{-1}F^\top B^\top BF) \leq C$. This ensures the temporally attended signal $BF$ spans the same $k$-dimensional space as $F$ up to well-conditioned scaling. We can estimate $\Lambda^{(A)}$ by applying PCA to $\widetilde{Z}^\top \widetilde{Z}$. Factors $F^{(B)}$ are obtained by regressing $\widetilde Z$ on the estimated loadings $\widehat{\Lambda}^{(A)}$.
We design the above methodology to exploit auxiliary information without compromising the structure of the target block. Conceptually, the target variables determine the latent directions that are relevant for the object of interest, while auxiliary variables, when appropriately weighted, can improve the precision with which these latent components are estimated. This separation allows the model to borrow strength across heterogeneous panels without allowing auxiliary information to redefine the target structure. pelger2024target_pca and our approach address related but distinct problems. Target PCA is developed to recover target factors and fill in missing target observations, which naturally leads to strong homogeneity assumptions and a limited role for auxiliary variables. Our framework instead targets forecasting environments in which auxiliary variables are plentiful, heterogeneous, and potentially more informative than the target block. This difference in objectives motivates a design that accommodates cross-sectional heterogeneity and enables transfer learning, allowing auxiliary data to improve estimation precision without redefining the target factor structure. The theoretical results in the next section formalize how these design choices lead to efficiency gains relative to target-only procedures whenever auxiliary information is informative.
The inferential theory in this section is derived under linear activation functions. Sections (ref) and (ref) provide conceptual and implementation details for nonlinear signals. Similarly to pelger2024target_pca, we work under a framework of general assumptions for approximate factor models.
We lay out the first set of assumptions, which we refer to as baseline assumptions.
The attention matrices $B$ and $A_z$ are treated as deterministic sequences that may depend on $(N_x+N_y,T)$ but are independent of the underlying randomness in $(F_t,e_t)$. This ensures that applying $(B,A_z)$ preserves the mixing and moment properties in Assumptions A.2--A.3.
Assumption A.1 ensures temporal attention does not distort the latent factor space: the transformation $B$ has bounded energy and preserves the rank-$k$ span of the factors up to a well-conditioned linear transformation. Assumptions A.2--A.4 correspond to the standard dependence and orthogonality conditions in approximate factor models; since $B$ and $A_z$ are deterministic linear operators with bounded operator norms, applying them preserves the mixing and moment properties of $(F_t,e_t)$. Assumption A.5 controls the behavior of the idiosyncratic component after cross-sectional and temporal attention, ensuring that the transformed noise does not explode in operator norm and satisfies the moment conditions needed for the CLTs. Assumption A.6 guarantees pervasiveness of the attention-weighted loadings, so that the cross-sectional law of large numbers holds after applying $A_z$. Assumption A.7 imposes spectral growth bounds on the attention matrix $A_z$, and ensures that the factor estimation error rate is $o_P(1)$.
The loadings and factors can be consistently estimated if $B$ and $A_z$ are chosen properly. \setcounter{thm}{0}
Theorem (ref) establishes consistency of the estimated loadings and factors under general cross-sectional and temporal attention matrices, and generalizes Theorem 1 of pelger2024target_pca to a fully attention-weighted setting.
The convergence rate $\bar{\alpha}$ is primarily governed by the cross-sectional attention matrix $A_z$ and depends on both the total magnitude of attention weights, captured by $\mathrm{tr}(A_z^\top A_z)$, and their concentration across variables, captured by $\|A_z^\top A_z\|_F^2$. In particular, Assumption A.7 imposes growth-rate restrictions $\mathrm{tr}(A_z^\top A_z)=\mathcal{O}(N^{\alpha})$ and $\|A_z^\top A_z\|_F^2=\mathcal{O}(N^{\beta})$, which together imply $\bar{\alpha}=\mathcal{O}(N^{\alpha+\beta-1})$. A crucial step in establishing consistency is Assumption A.7, which controls the dispersion of cross-sectional attention weights. This condition is easily satisfied in several empirically relevant cases. First, when attention is normalized and evenly spread across variables---for example, when each column of $A_z$ has squared Euclidean norm of order $1/(N_x+N_y)$---we obtain $\mathrm{tr}(A_z^\top A_z)=\mathcal{O}(1)$ and $\|A_z^\top A_z\|_F^2 \ll N_x+N_y$, ensuring $\bar{\alpha}=o(1)$. Second, when cross-sectional attention is sparse and concentrates on a subset of $m \ll N_x+N_y$ variables with unit column norm, we have $\mathrm{tr}(A_z^\top A_z)\approx m$ and $\|A_z^\top A_z\|_F^2\approx m$, which still yields consistency provided $m$ grows sufficiently slowly relative to $N_x+N_y$. In contrast, if attention weights are excessively concentrated so that Assumption A.7 is violated, the convergence rate deteriorates and consistency of the factor and loading estimators may fail. The temporal attention matrix $B$ enters the rate only through the factor $\|B\|_F^2/T$. Under mild regularity conditions---such as boundedness, smoothness, or sparsity of temporal attention---we naturally have $\|B\|_F^2/T=\mathcal{O}(1)$, so temporal attention does not affect the consistency rate.
To derive asymptotic normality for the target component $Y$, it is not enough to know that the estimated factor space is overall consistent (Theorem (ref)). We must also identify which coordinates of the global $k$-dimensional factor space correspond to the $Y$-strong factors that drive the target block. This requires additional structural conditions. Assumptions B.1--B.3 and Lemma (ref) establish identification conditions formally and allow us to isolate the $Y$-strong subspace for inference.
Recall the attention-weighted panel in (ref):
Here $\widetilde{Z}_t \in \mathbb{R}^{N_x+N_y}$ denotes the $(N_x+N_y)$-dimensional cross section at time $t$, obtained by stacking the $X$- and $Y$-components. $F_t^{(B)} \in \mathbb{R}^k$ are the common factors, $\Lambda_x^{(A)} \in \mathbb{R}^{N_x \times k}$, $\Lambda_y^{(A)} \in \mathbb{R}^{N_y \times k}$ are the attention-weighted loadings, and $ e_t^{(A,B)}$ are idiosyncratic terms.
Partition the factor vector and loadings as
where we decompose $k=k_{y_s}+k_R$, with $k_{y_s}$ denoting the number of $Y$-strong factors and $k_R$ collecting the remaining factors. $F_{y,t}^{S} \in \mathbb{R}^{k_{y_s}}$ denotes the $Y$-strong factors, $F_{t}^{R}$ collects all remaining factors ($X$-only, and shared but weak in $Y$), $\Lambda_{y_s}^{(A)} \in \mathbb{R}^{N_y \times k_{y_s}} $ are the $Y$-loadings on $Y$-strong factors, $\Lambda_{x,y_s}^{(A)} \in \mathbb{R}^{N_x \times k_{y_s}}$ are the $X$-loadings on $Y$-strong factors, $\Lambda_{y,R}^{(A)} \in \mathbb{R}^{N_y \times k_{R}}$ and $\Lambda_{x,R}^{(A)} \in \mathbb{R}^{N_x \times k_{R}} $ collect the remaining loadings on $Y$ and $X$, respectively. Intuitively, $Y$-strong factors are those with non-vanishing cross-sectional signal in the $Y$ block, whereas the remaining factors are either $X$-only or have vanishing (weak) loadings in $Y$. Define the block-specific loading covariance matrices \[ \Sigma_{y_s,y}^{(A)} \equiv \lim_{N_{y,\mathrm{eff}}\to\infty} \frac{1}{N_{y,\mathrm{eff}}} \Lambda_{y_s}^{(A)\top}\Lambda_{y_s}^{(A)}, \qquad \Sigma_{y_s,x}^{(A)} \equiv \lim_{N_{x,\mathrm{eff}}\to\infty} \frac{1}{N_{x,\mathrm{eff}}} \Lambda_{x,y_s}^{(A)\top}\Lambda_{x,y_s}^{(A)}. \]
Let \[ P_x \equiv
\quad P_y \equiv
\] be the coordinate projections onto the original $X$ and $Y$ blocks. The effective cross-sectional sizes are
When $A_z$ is block-diagonal, we have $ N_{\text{x,eff}} \equiv \text{tr}(A_1^\top A_1)$ and $N_{\text{y,eff}} \equiv \text{tr}(A_2^\top A_2)$.
We impose the following additional assumptions, which we refer to as strong-factor structure and attention scaling conditions:
Assumptions B.1--B.3 formalize the structure required to identify the $Y$-strong factor subspace within the global factor space after attention weighting. Assumption B.1 states that the $Y$ block of the attention-weighted panel contains $k_{y_s}$ strong factors whose associated eigenvalues grow at rate $N_{\text{y,eff}}$, while all remaining factors contribute only lower-order eigenvalues to the $Y$-block covariance. This restriction concerns only the residual factor directions within the $Y$ block and does not preclude the auxiliary $X$ block from carrying non-negligible signal about the $Y$-strong factors (Assumption B.3). Although Assumption B.1 normalizes the strength of the $Y$-strong factors using $N_{y,\mathrm{eff}}$, the convergence rates in Theorems (ref)--(ref) are governed by $N_{\mathrm{eff}}$, reflecting the fact that factor estimation exploits information from the full attention-weighted panel, while identification of the $Y$-strong subspace relies solely on the $Y$ block. In contrast to pelger2024target_pca, where both identification and estimation precision are driven solely by the size of the target block, our framework separates the two roles: the $Y$ block alone identifies the $Y$-strong factor subspace, while the full attention-weighted panel determines the estimation rate via $N_{\mathrm{eff}}$.
Assumption B.2 ensures that the cross-sectional attention matrix does not distort the effective cross-sectional sizes, so that $N_{\text{x,eff}}$ and $N_{\text{y,eff}}$ grow proportionally to $N_x$ and $N_y$, respectively, so attention redistributes information across units without concentrating mass on a vanishing subset or artificially amplifying cross-sectional signal. Assumption B.3 formalizes a transfer-learning setting in which the target block $Y$ identifies the Y-strong factor subspace, while the auxiliary block $X$ provides additional but incomplete information that improves estimation efficiency without altering the target-specific nature of the factors. Assumption B.4 is a standard orthogonality condition ensuring that the cross-sectional variation used to estimate factors is asymptotically uncorrelated with the time-series variation used to estimate loadings. This condition eliminates interaction terms in the asymptotic distribution of the estimated common component and simplifies inference in the mixed regime.
Our Assumption B.3 strengthens the “no full spanning” condition (Condition G.1) of pelger2024target_pca by additionally requiring the auxiliary panel $X$ to carry non-negligible signal in the $Y$-strong factor directions. While pelger2024target_pca only require that $X$ does not fully span the $Y$-strong space, a minimal condition needed for identification when the target panel $Y$ may be partially observed, our transfer-learning setting explicitly focuses on efficiency gains from auxiliary information. Consequently, we allow $X$ to be informative, but not fully identifying, for the $Y$-strong factors. Target PCA arises as a special case of our framework when $X$ carries no signal in the $Y$-strong directions.
Let $\Sigma_Z^{(A)} = \mathbb{E} \lbrack \widetilde{Z}_t \widetilde{Z}_t^\top \rbrack$ denote the population covariance of the attention-weighted panel, and let $\mathcal{S}_Y$ denote the column space of $\Lambda_{y_s}^{(A)}$:
Lemma (ref) explains how the target block $Y$ identifies economically relevant factors within the globally estimated factor space. While PCA applied to the attended panel $\widetilde Z$ consistently recovers the $k$-dimensional factor space, it does not by itself determine which factor directions drive variation in $Y$. Part (a) shows that, under block-specific strength conditions, the $Y$-block covariance exhibits exactly $k_{y_s}$ eigenvalues diverging at rate $N_{y,\mathrm{eff}}$, ensuring that the associated eigenspace coincides with the span of the $Y$-strong loadings. Part (b) then establishes that the corresponding factor coordinates are unique up to an orthogonal rotation within the $k_{y_s}$-dimensional subspace. Together, these results provide the key identification step needed for inference on the target component.
Theorem (ref) shows the asymptotic distribution of estimated $Y$-strong factors and loadings.
$\Omega_{y,i}^{(A,B)}$ and $\Xi_{t}^{(A,B)} $ are long-run covariance matrices, allowing for temporal and cross-sectional dependence induced by attention weighting. The identification of Theorem (ref) is ensured by Assumptions B.1--B.3 and Lemma (ref), which separates the $Y$-strong subspace from the rest of the factor space. Although the $Y$-strong factors are identified by the $Y$-block covariance, their estimation exploits auxiliary information from $X$ through joint regression, leading to $\sqrt{N_{\text{eff}}}$ convergence under non-negligible cross-block signal.
\paragraph{Comparison with classical PCA and Target PCA.} The growth conditions in Theorem (ref) imply $T=o(N^{1-(\alpha+\beta)})$.These conditions are a natural extension of those in classical approximate factor models. In bai2003inferential, consistency and inference are obtained under the regime $T=o(N)$, reflecting the requirement that effective cross-sectional information dominates the time dimension. Our conditions recover this classical case when the cross-sectional attention matrix is diffuse, so that $\mathrm{tr}(A_z^\top A_z)\asymp N$ and $\|A_z^\top A_z\|_F^2=\mathcal O(N)$, implying $\bar\alpha=\mathcal O(1/N)$ and hence $T\bar\alpha\to0$ reduces to $T=o(N)$. When attention concentrates mass on a lower-dimensional subset of units, the effective cross-sectional size $N_{\mathrm{eff}}=\mathrm{tr}(A_z^\top A_z)$ grows more slowly than $N$, and the admissible growth rate of $T$ adjusts accordingly. This trade-off is analogous to the bias--variance balance induced by regularization or projection in high-dimensional estimation and is intrinsic to attention-weighted designs. Relative to pelger2024target_pca, which enforces a fixed projection structure and primarily exploits the target block to recover strong factors, our framework allows auxiliary units to contribute at the $\sqrt{N_{\mathrm{eff}}}$ scale without altering identification, yielding inference conditions that interpolate smoothly between classical PCA and targeted factor extraction. In this sense, the proposed growth conditions are not restrictive but rather endogenously determined by the degree of cross-sectional concentration induced by attention.
Define the true and estimated $Y$-strong common component as \[ C_{y,i,t}\equiv \Lambda_{y_s,i}^{(A)\top}F_{y,t}^S, \qquad \widehat C_{y,i,t}\equiv \widehat\Lambda_{y_s,i}^{(A)\top}\widehat F_{y,t}^S. \] Then \[ \widehat C_{y,i,t}-C_{y,i,t} = \underbrace{\Lambda_{y_s,i}^{(A)\top}\Big(\widehat F_{y,t}^S-(H_{y_s}^{(A)\top})^{-1}F_{y,t}^S\Big)}_{\Delta_{F,it}} + \underbrace{\Big(\widehat\Lambda_{y_s,i}^{(A)}-H_{y_s}^{(A)}\Lambda_{y_s,i}^{(A)}\Big)^\top (H_{y_s}^{(A)\top})^{-1}F_{y,t}^S}_{\Delta_{\Lambda,it}} +r_{i,t}, \] with $r_{i,t}=o_p(N_{\mathrm{eff}}^{-1/2}+T^{-1/2})$.
Let $V_{\Lambda,y,i}^{(A,B)}$ and $V_{F,t}^{(A,B)}$ be the asymptotic covariance matrices in Theorem (ref). Define \[ \sigma^2_{C,it,F}\equiv \Lambda_{y_s,i}^{(A)\top}V_{F,t}^{(A,B)}\Lambda_{y_s,i}^{(A)}, \qquad \sigma^2_{C,it,\Lambda}\equiv \big((H_{y_s}^{(A)\top})^{-1}F_{y,t}^S\big)^\top V_{\Lambda,y,i}^{(A,B)} \big((H_{y_s}^{(A)\top})^{-1}F_{y,t}^S\big). \]
Theorem (ref) establishes the asymptotic distribution of $C_{y,i,t}$ for three different regimes that depend on the relative growth of $N_{\text{eff}}$ and $T$.
The asymptotic behavior of the $Y$-common component reflects the relative informativeness of the cross-sectional and time-series dimensions after attention weighting.
In regime (i), where $N_{\mathrm{eff}}/T\to0$, the time dimension is large relative to the effective cross section, so factor estimation error dominates the asymptotic distribution of the common component. In regime (ii), where $T/N_{\mathrm{eff}}\to0$, the effective cross section is large relative to the time dimension, and loading estimation error is the primary source of uncertainty. Regime (iii) covers scenarios in which neither dimension is asymptotically dominant, and both factor and loading estimation errors contribute to the limiting distribution. This mixed regime naturally accommodates balanced panels in which $N_{\mathrm{eff}}$ and $T$ grow at comparable rates.
Sections (ref) and (ref) developed a theoretical framework for extracting low-dimensional latent representations from attention-weighted panels under linear signal structures. This framework can be viewed as a solution to a representation learning problem, in which the attended data $\widetilde Z$ are projected onto a low-dimensional latent space that captures their dominant common components.
In many empirical applications, however, linear representations may be restrictive. A growing literature in macroeconomics and finance documents substantial nonlinearities in the mapping from high-dimensional predictors to latent economic states and forecasts cheng2015forecasting, kelly2021autoencoder, goulet2025panel. Rather than abandoning the linear framework developed above, we embed it within a more general representation learning architecture that allows for nonlinear signal extraction while preserving the same latent-variable interpretation.
Recall that the target and auxiliary datasets have the following signal plus noise structure:
The “attended” inputs were defined as
Recall that in the linear case the factor model for $\widetilde{Z}$ is:
Let $\widehat D^{(A)} \coloneqq \mathrm{diag}(\lambda_1^{(A)},\ldots,\lambda_k^{(A)})$ denote the diagonal matrix collecting the largest $k$ eigenvalues of $\widetilde Z^\top \widetilde Z /(N_x+N_y)$. These eigenvalues capture the relative strength of the attention-weighted common components, analogously to standard PCA. Let $S=\mathrm{diag}(s_1,\ldots,s_k)$ denote the corresponding singular values, so that \[ \widehat D^{(A)} = \frac{S^2}{N_x+N_y}. \]
Equivalently to the standard PCA formulation in (ref), a singular value decomposition (SVD) of $\widetilde Z$ yields
where $\widehat P \in \mathbb{R}^{T\times k}$ and $\widehat Q \in \mathbb{R}^{(N_x+N_y)\times k}$ contain the left and right singular vectors associated with the top $k$ singular values collected in the diagonal matrix $\widehat{D}^{(A)}$.
This representation is equivalent to a one-layer linear autoencoder with $k$ latent units. Specifically, letting \[ \mathcal E_\theta(\widetilde Z_t) = b^{(0)} + W^{(0)} \widetilde Z_t \in \mathbb{R}^k \] denote the encoding map $\mathcal E_\theta:\mathbb{R}^{N_x+N_y}\to\mathbb{R}^k$ that extracts a $k$-dimensional summary of the attended panel $\widetilde Z_t$.
The reconstruction is given by
where $W^{(0)} \in \mathbb{R}^{k\times (N_x+N_y)}$, $W^{(1)} \in \mathbb{R}^{(N_x+N_y)\times k}$, $b^{(0)} \in \mathbb{R}^k$ are weight matrices, and $b^{(1)} \in \mathbb{R}^{N_x+N_y}$ are bias vectors.
The parameters $(b^{(0)}, b^{(1)}, W^{(0)}, W^{(1)})$ are estimated by solving
where $\iota$ is a $T\times1$ vector of ones.
The next proposition, adapted from kelly2021autoencoder, establishes the equivalence between the linear encoder in (ref) and the PCA estimator introduced in Section (ref).
Proposition (ref) shows that a one-layer linear autoencoder with $k$ hidden units is equivalent to a linear factor model with $k$ latent factors. Up to an invertible linear transformation, the estimated latent factors and loadings coincide with those obtained from attention-weighted PCA. In particular, the estimated factors and loadings may be written as \[ \widehat F^{(B)} = \widehat P R^{-1}(N_x+N_y)^{1/2}, \qquad \widehat \Lambda^{(A)} = \widehat Q R (N_x+N_y)^{1/2}, \] which matches the normalization used in Section (ref). In other words, in the linear case, $\mathcal E_\theta(\cdot)$ coincides (up to rotation and scaling) with the PCA factor extraction map.
Viewed through the lens of representation learning, the linear autoencoder analyzed above induces an affine encoding map $\mathcal E_{\theta}:\mathbb{R}^{N_x+N_y}\to\mathbb{R}^{k}$ that extracts a $k$-dimensional low-rank representation from the attended panel $\widetilde Z$. This encoding map coincides, up to rotation, with PCA-based factor extraction in the linear case. A natural extension is to enlarge the class of admissible encoding maps by allowing for nonlinear transformations, while preserving the same low-dimensional bottleneck and representation-learning structure.
Autoencoder models are therefore more general than linear factor models, as they permit nonlinear signal extraction through compositions of nonlinear transformations of $\widetilde Z$. In particular, autoencoder architectures with $M$ hidden layers can be written recursively as follows.
Let $K^{(m)}$ denote the number of neurons in layer $m=1,\ldots,M$. For each layer $m$, let $\widetilde Z^{(m)} \in \mathbb{R}^{K^{(m)}}$ denote the vector of neuron outputs, with components $\widetilde Z^{(m)} = (\widetilde Z^{(m)}_1,\ldots,\widetilde Z^{(m)}_{K^{(m)}})^\top$. The input layer is given by $\widetilde Z^{(0)} \equiv \widetilde Z$.
For $m \ge 1$, the output of layer $m$ is generated recursively according to
where $g(\cdot)$ is an elementwise nonlinear activation function, $W^{(m-1)} \in \mathbb{R}^{K^{(m)} \times K^{(m-1)}}$ is a matrix of weight parameters, and $b^{(m-1)} \in \mathbb{R}^{K^{(m)}}$ is a vector of bias parameters. By construction, $\widetilde Z^{(m)} \in \mathbb{R}^{K^{(m)}}$ and $\widetilde{Z}^{(m-1)} \in \mathbb{R}^{K^{(m-1)}}$.
The nonlinear encoding map is defined as the composition of the layer-wise transformations, \[ \mathcal E_{\theta}(\widetilde Z) \;\equiv\; \widetilde Z^{(M)} = \bigl(\phi_{M,\theta} \circ \cdots \circ \phi_{1,\theta}\bigr)(\widetilde Z), \qquad \phi_{m,\theta}(u)=g\!\left(b^{(m-1)}+W^{(m-1)}u\right), \] so that $\mathcal E_{\theta}:\mathbb{R}^{N_x+N_y}\to\mathbb{R}^{k}$ denotes the terminal output of the $M$-layer encoder and extracts a $k$-dimensional low-rank representation from the attended panel $\widetilde Z$. Here $\circ$ denotes function composition ($(f \circ g)(x) = f(g(x))$).
When the activation function satisfies $g(x)=x$ and the encoder has a single hidden layer ($M=1$), the encoding map $\mathcal E_{\theta}$ reduces to the linear case and coincides, up to rotation, with attention-weighted PCA. More general choices of $g(\cdot)$ enlarge the class of admissible encoding maps, allowing $\mathcal E_{\theta}(\widetilde Z)$ to capture nonlinear structure in $\widetilde Z$ while retaining the same low-dimensional bottleneck interpretation. Figure (ref) provides a schematic summary of this nesting, highlighting that the linear and nonlinear specifications differ only in the choice of the transformation $g(\cdot)$ governing the mapping of the attended panel $\widetilde Z$. We implement the nonlinear encoding map $\mathcal E_\theta$ using a Transformer-based architecture, which parameterizes the activation $g(\cdot)$ in (ref). Specifically, the attended inputs $\widetilde Z$ are processed by a stack of feedforward layers of the form (ref), with attention mechanisms used to model complex cross-sectional and temporal interactions. Implementation details of the Transformer architecture are provided in Section (ref).
We use this architecture for representation learning because of its widespread adoption and demonstrated effectiveness in modern large-scale sequence models. In particular, Transformer encoders are the core component of recent state-of-the-art language models trained via instruction tuning and reinforcement learning from human feedback ouyang2022training. We provide the details on the Transformer and implementation nuances in Section (ref).
The architecture described above enables us to construct a nonlinear factor extractor in which attention captures adaptive contextual dependencies and the feedforward stack captures complex hierarchical interactions. We do not develop a full inferential theory for the nonlinear encoder, as the analysis of deep nonlinear architectures remains challenging. Instead, our contribution is to show that the proposed nonlinear model is a direct generalization of the attention-weighted linear factor model studied in Sections (ref) and (ref), sharing the same inputs, latent dimension, and reconstruction-based objective.
In this section, we describe how we implement MPTE in finite samples for the simulation and empirical analyses. Our goal is to implement the estimator introduced in Sections (ref)--(ref), while fixing the data preprocessing and input normalization conventions used throughout the remainder of the paper. The implementation nests the linear MPTE studied in the theoretical sections and extends it to nonlinear signal extraction and mixed-frequency datasets without requiring resampling or frequency-specific preprocessing.
An assumption imposed in Sections (ref)--(ref) is that $T_x = T_y = T$, which is convenient for the theoretical analysis as it allows the panels to be concatenated as $Z = [X \quad Y]$ and standard matrix operations to be applied. This assumption is, however, restrictive in empirical applications, particularly in mixed-frequency forecasting settings where auxiliary variables are observed at higher frequencies than the target. Traditional approaches, such as MIDAS, address this issue through manual alignment or aggregation across frequencies, which may entail information loss.
In the empirical setting, we relax the restriction $T_x = T_y$ by embedding both the target and auxiliary datasets into a unified sequence representation indexed by calendar time. This representation allows the model to aggregate information across frequencies via attention, without resampling or pre-specified aggregation weights. While the formal theoretical guarantees developed in Sections (ref)--(ref) do not extend to the mixed-frequency setting, the resulting estimator remains well defined and can be implemented straightforwardly in finite samples.
We represent the observations from the target panel $Y$ and the auxiliary panel $X$ as a single ordered sequence indexed by calendar time. Each sequence entry corresponds to a single observed variable at a given time point and may originate from either of the two datasets. This representation allows variables sampled at heterogeneous frequencies to be embedded in a unified temporal structure, while preserving their original observation times.
Let $\{(v_\ell,t_\ell)\}_{\ell=1}^L$ denote the ordered sequence of observed variable--time pairs, where $\ell$ indexes the position of the observation in the sequence, $v_\ell\in\{1,\ldots,N_x+N_y\}$ is the variable identifier associated with position $\ell$, $t_\ell$ is the calendar timestamp associated with the observation at position $\ell$, and $L$ is the total number of sequence entries. The sequence is ordered so that $t_1 \le \cdots \le t_L$, but observations may originate from different sampling frequencies, and multiple variables may appear at the same timestamp. The ordering among observations sharing the same time index is arbitrary but fixed.
For each sequence entry $\ell$, let $r_{v_\ell,t_\ell}$ denote the raw observed value for the variable $v_\ell$ at time $t_\ell$. We standardize all raw numerical variables using sample moments computed over the available time observations for each variable,
where $\mu_{v_\ell}$ and $\sigma_{v_\ell}$ are the sample mean and standard deviation of variable $v_\ell$. In the simulation and empirical analyses, we compute these normalization statistics using only the in-sample portion of the data and apply them unchanged to the out-of-sample period, in order to avoid information leakage. This standardization step ensures comparability across series and prevents scale differences from dominating the attention mechanism. Standardizing inputs is a common practice when training neural network--based models, as it improves numerical stability and optimization behavior goodfellow2016deep.
We then map each standardized observation into a sequence-level input representation that jointly encodes its numerical value, variable identity, and sampling frequency. Specifically, define the embedding map
where $e^{(\mathrm{var})}_{v_\ell}$ is a learned embedding associated with variable $v_\ell$, $e^{(\mathrm{freq})}_{f_\ell}$ is a learned embedding associated with the sampling frequency $f_\ell$, $f_\ell$ denotes the sampling frequency associated with observation $\ell$ (e.g., monthly or quarterly), with $e^{(\mathrm{var})}_{v_\ell} \in \mathbb{R}^{d_{\mathrm{var}}}$ and $e^{(\mathrm{freq})}_{f_\ell} \in \mathbb{R}^{d_{\mathrm{freq}}}$, so that $d_{\mathrm{in}} = 1 + d_{\mathrm{var}} + d_{\mathrm{freq}}$. We obtain the embeddings $e^{(\mathrm{var})}_{v}$ and $e^{(\mathrm{freq})}_{f}$ from learnable embedding tables, and we update those parameters jointly with all other MPTE parameters during training. Conceptually, these embeddings act as learned, low-dimensional parameter vectors associated with categorical identifiers, analogous to variable-specific latent characteristics and frequency-specific effects. In the data matrix passed to the encoder, each $\phi(r_{v_\ell,t_\ell}^{*}, v_\ell, f_\ell)$ corresponds to one row, and stacking these rows yields the input sequence.
As a further step, we project the concatenated representation into the model dimension before applying attention. This projection learns a joint representation of the standardized value, variable identity, and sampling frequency, and allows the attention mechanism described in Section (ref) to operate on latent features rather than on raw inputs. Following standard Transformer architectures vaswani2017attention, we define
where $d_{\mathrm{in}}$ denotes the dimension of the concatenated representation $\phi(\cdot)$, $d_{\mathrm{model}}$ is the fixed internal dimension of the Transformer encoder that forms part of the MPTE specification, and the projection matrix $W_{\mathrm{proj}}$ is estimated jointly with all other encoder parameters.
Although the sequence entries are ordered by their calendar timestamps, MPTE has no intrinsic notion of temporal ordering without an explicit encoding. We therefore explicitly incorporate temporal information through a temporal encoding. Specifically, we define
where $t_\ell$ denotes a numerical time index obtained from the corresponding calendar timestamp. Here $\mathrm{TE}(t_\ell)\in\mathbb{R}^{d_{\mathrm{model}}}$ is a deterministic sinusoidal temporal encoding defined componentwise as
as in vaswani2017attention. We add the temporal encoding rather than concatenating it so that time enters the representation as a time index governing relative relationships across sequence entries, rather than as an additional observed feature.
Stacking the sequence entries yields the encoder input matrix
This sequence-based representation replaces, at the implementation level, the wide stacked panel $[ X \quad Y ]$ used in the theoretical analysis with a corresponding ordered collection of observations that is amenable to attention-based aggregation when sampling frequencies differ.
Starting from the embedded sequence $Z$, we pass it through a Transformer encoder following the standard architecture of vaswani2017attention. The Transformer encoder can consist of a stack of identical encoder layers, each applying an attention operation followed by a feedforward transformation. In the simplest case, the encoder contains a single layer.
Within each encoder layer, we apply the attention mechanism to the input sequence to produce an attended representation,
where $\mathcal{A}_{\theta_A}(\cdot)$ denotes a data-dependent attention operator parameterized by learnable weight matrices. These parameters correspond to the linear transformations defining queries, keys, and values, as described in Section (ref). This operation is the nonlinear analogue of the attention-based reweighting studied in the theoretical analysis and induces both temporal and cross-sectional aggregation across sequence entries. Figure (ref) illustrates the distinction between the wide-panel representation used in the theoretical analysis and the long-sequence representation used in implementation. The former relies on a common time index to concatenate panels horizontally, while the latter organizes mixed-frequency observations as an ordered sequence of variable–time tokens that are embedded and processed via attention.
We then pass the attended representation $\widetilde Z^{\mathrm{att}}$ through a feedforward transformation with elementwise nonlinear activation,
which corresponds to the layerwise transformation in (ref). Stacking multiple encoder layers iterates this attention–nonlinearity composition and yields a nonlinear encoding map $\mathcal E_{\theta}(\cdot)$ as defined in Section (ref). The final encoder output $\mathcal E_{\theta}(Z) = \widetilde Z^{(M)}$ provides a low-dimensional representation that captures higher-order and context-dependent interactions across variables, sampling frequencies, and time.
In the theoretical analysis, we decomposed attention into separate temporal ($B$) and cross-sectional ($A_z$) operators to clarify identification and asymptotic properties. However, in the implementation, these effects are learned jointly within MPTE through a single data-dependent attention mechanism $\mathcal{A}_{\theta_A}(\cdot)$ operating on the ordered sequence $Z$. The resulting attention weights endogenously determine how information is reweighted across time and across variables.
When the nonlinear activation $g(\cdot)$ is replaced by the identity function, the encoding map $\mathcal E_{\theta}$ reduces to the linear MPTE estimator analyzed in Sections (ref)--(ref), and representation extraction coincides, up to rotation, with PCA estimation on attention-weighted inputs. More generally, the encoder implements a nonlinear factor extractor that preserves the estimator architecture underlying the linear theory while allowing for context-dependent signal construction. Depending on how the encoded representation $\mathcal E_{\theta}$ is used downstream, the same framework can be applied to regression, classification, or related prediction tasks.
For the purposes of this paper, and in order to illustrate how mixed-frequency information can enhance forecasting performance, we focus on a regression setting. In the simulation and empirical analyses, we construct forecasts by applying a linear mapping to the encoder output $\mathcal E_{\theta}(Z)$, producing continuous-valued predictions for the target variables at the desired forecast horizons. We train MPTE by jointly estimating all model parameters, including the variable- and frequency-embedding tables, the projection layer, the Transformer encoder parameters, and the prediction head, using gradient-based optimization.
We select architectural and training hyperparameters for both the simulation and empirical exercises using Bayesian optimization, and we document the corresponding optimization procedures and selected values in Appendix (ref).
To assess the finite-sample performance of MPTE under controlled forms of nonlinearity and mixed-frequency information, we consider a data-generating process (DGP) inspired by lin2024ijf. We model the dynamics of the latent factors using a linear VAR(2) process:
The latent factor process $\{F_t\}$ is generated at the high frequency. Each low-frequency index $t'$ is associated with the high-frequency time $t = r t'$, corresponding to the end of the $t'$-th low-frequency period, where $r$ denotes the ratio between the high- and low-frequency sampling intervals (e.g.\ $r=3$ for monthly and quarterly frequencies). We rescale the coefficient matrices $(\Phi_1,\Phi_2)$ to ensure stability of the latent VAR(2) process, with the spectral radius of the associated companion matrix bounded away from unity. We set the innovation covariance to $\Sigma_F = 0.5\, I_q$ and discard an initial burn-in period.
Conditional on the latent factor process $\{F_t\}$, we generate two panels of observables at different sampling frequencies, namely a high-frequency panel $\{X_t\}$ following a VAR($L_x$) model indexed by $t$, and a low-frequency panel $\{Y_{t'}\}$ following a VAR($L_y$) model indexed by $t'$:
We rescale the autoregressive coefficient matrices $\{A_\ell\}_{\ell=1}^{L_x}$ and $\{C_\ell\}_{\ell=1}^{L_y}$ so that the spectral radius of the associated companion matrices does not exceed pre-specified block-specific thresholds, allowing for different degrees of persistence across high- and low-frequency variables. We generate the factor loading matrices $\Lambda_{x,j}$ and $\Lambda_{y,j}$ using Almon polynomial lag weights to induce decaying factor effects across lags smoothly. We also scale the noise variances to ensure comparable signal-to-noise ratios across simulation designs.
To control the degree of nonlinearity in the factor–observable relationship, we specify the latent transformation $g(\cdot)$ as follows:
where $\lVert\cdot\rVert$ denotes the Euclidean norm, and $\sigma_k$ denotes the sample standard deviation of the $k$-th RBF component computed over time, so that each RBF feature is standardized to have unit variance. The centers $\{c_k\}_{k=1}^K \subset \mathbb{R}^q$ of the radial basis function (RBF) are drawn i.i.d.\ from $\mathcal{N}(0,I_q)$, and the bandwidth parameter $\gamma$ is set using a median-distance heuristic based on the centers.
We consider three simulation regimes. In the linear design, we set $g(\cdot)$ equal to the identity map. In the nonlinear designs, we use the RBF specification in (ref) and vary the number of basis functions $K$ to control the degree of nonlinearity. Specifically, we set $K=6$ in the mildly nonlinear case and $K=12$ in the highly nonlinear case, where $K$ denotes both the number of RBF centers and the resulting number of basis functions, holding all other aspects of the DGP fixed across regimes.
Across all simulation regimes, we generate 5{,}000 high-frequency observations, using the first 4{,}000 after burn-in for training, with 10% of the sample reserved for validation and the remaining 1{,}000 for out-of-sample evaluation. We report results for the first low-frequency target series $Y_1$. Since all low-frequency variables are generated from the same DGP and differ only through their factor loadings and idiosyncratic shocks, forecasting other targets yields qualitatively similar results\footnote{The implementation used for the simulation exercises in this section, as well as for the empirical analysis in the next section, is based on a unified codebase available at \url{https://github.com/Alessiobrini/mixed-panels-transformer-encoder}.}.
We compare the performance of MPTE against two benchmark models: (i) a univariate autoregressive model (AR), for which we select the order using the Bayesian Information Criterion (BIC), and (ii) an unrestricted linear MIDAS specification in which high-frequency predictors enter through unrestricted distributed lags, without nonlinear transformations or parametric lag-weighting functions. In addition, we evaluate the following ablation variants of MPTE to isolate the contribution of individual architectural components:
We report forecasting performance using root mean squared error (RMSE), mean absolute error (MAE), and directional accuracy (DA), defined as the fraction of periods in which the predicted and actual values of the target series share the same sign. We train all MPTE specifications and ablation variants using automated hyperparameter optimization. We report the full details on the optimization procedure and search spaces used in the simulation experiments in Appendix (ref).
Table (ref) summarizes forecasting performance across linear, mildly nonlinear, and highly nonlinear designs. In the linear setting, MPTE does not outperform MIDAS and yields comparable or slightly inferior accuracy, which is expected since MIDAS is itself a linear model and is well suited for learning purely linear relationships without introducing unnecessary model complexity. In contrast, MPTE delivers clear gains over both MIDAS and AR in the mildly and highly nonlinear designs. These results underscore the benefits of combining nonlinear signal extraction with mixed-frequency information.
The ablation results provide additional insight into the role of individual architectural components. Removing nonlinear transformations (AB1) deteriorates performance in the nonlinear designs but has limited impact in the linear case, confirming that nonlinear feature construction is beneficial primarily when warranted by the underlying DGP. Similarly, disabling attention (AB2) or both attention and nonlinearity (AB3) leads to notable performance losses in the nonlinear settings, indicating that attention-based aggregation plays a key role in selectively combining information across time and variables when signals are nonlinear. In the linear design, however, attention does not systematically improve performance and may even introduce mild overfitting, consistent with the fact that linear dynamics can be adequately captured by simpler aggregation schemes.
Temporal encoding also proves important across all designs. The absence of temporal encoding (AB4) leads to inferior performance in both linear and nonlinear settings, underscoring the importance of explicitly encoding temporal order when applying Transformer-based models to sequential data. Finally, restricting the model to low-frequency information only (AB5) results in higher forecast errors relative to the full MPTE specification, reflecting the efficiency gains from incorporating high-frequency predictors that share common latent factors with the target series. This finding mirrors empirical macroeconomic settings in which monthly indicators provide valuable information for forecasting quarterly aggregates through shared underlying economic drivers.
To evaluate the empirical performance of MPTE in a realistic mixed-frequency forecasting environment, we apply the model to a large quarterly–monthly macroeconomic dataset. Mixed-frequency structures arise naturally in macroeconomic applications, where key aggregates are observed at low frequency while a broad set of indicators is available at higher frequency. We therefore consider the macroeconomic database of DataMcCracken (FRED-QD and FRED-MD), which provides a comprehensive panel of U.S.\ macroeconomic time series at quarterly and monthly frequencies.
We study the period from 1959:Q1 to 2025:Q1, corresponding to 1959:M1–2025:M3, using the current vintage as of 2025:M3. From this database, we select thirteen quarterly variables as forecast targets: GDPC1, GPDIC1, PCECC96, DPIC96, OUTNFB, UNRATE, PCECTPI, PCEPILFE, CPIAUCSL, CPILFESL, FPIx, EXPGSC1, and IMPGSC1. The complete list of monthly and quarterly regressors used in the empirical analysis, together with brief descriptions and their category, is reported in Appendix (ref).
We conduct the forecasting exercise at a quarterly frequency. For each target observation, we construct an input sequence for the model comprising all available monthly and quarterly observations within a fixed two-year context window, ending at the most recent observed quarter preceding the forecasted quarter. This context window defines the information set supplied to MPTE and may include both high- and low-frequency variables, as described in Section (ref). The task is to predict the next quarterly realization of the target series. The fixed two-year context window can be viewed as a finite-sample analogue of the temporal operator $B$ studied in the theoretical analysis, in that it restricts the attention mechanism to operate on a local time neighborhood while allowing the model to learn how to allocate weighs within that neighborhood.
As in the simulation exercises, we split the sample sequentially to preserve the temporal ordering of the data. We use the first 80% of the available observations for model training, and we reserve the final 10% of this training sample for validation. We hold out the remaining 20% of the data for out-of-sample evaluation. As in Section (ref), we select model hyperparameters using automated optimization conducted separately for each target variable and model specification (see Appendix (ref)).
We compare MPTE with the same benchmark models used in the simulation study, namely a univariate AR model and an unrestricted linear MIDAS specification. In addition, we include other competing models: OLS, eXtreme Gradient Boosting (XGB) chen2016xgboost, and a feedforward neural network (NN). Since OLS, XGB, and NN do not natively support mixed-frequency inputs, we estimate these models using only quarterly predictors. For comparability across models, we aggregate the monthly variables to the quarterly frequency by taking the last value of the quarter, so that all competing models are trained on a common quarterly information set.
Consistent with the simulation analysis, we evaluate forecasting performance using the same metrics as in Section (ref): RMSE, MAE, and DA. We construct all subsamples based on forecast timestamps to assess the stability of forecasting performance around the COVID-19 period. The full evaluation period spans from June 2012 to March 2025. We further split this period into a pre-COVID subsample, covering forecasts dated up to June 2019, and a COVID and post-COVID subsample, covering forecasts dated after June 2019. We apply these splits consistently across all models and performance metrics. In addition to point forecast accuracy metrics, we report pairwise forecast comparison tests based on the Diebold--Mariano test, as well as Model Confidence Set (MCS) results in Appendix (ref).
Tables (ref) and (ref) report out-of-sample forecasting performance at the individual target level, grouped according to whether MPTE achieves the lowest RMSE over the full sample period. Table (ref) focuses on target series for which MPTE attains the lowest RMSE, while Table (ref) reports results for series where alternative models outperform MPTE. Table (ref) provides a complementary summary by counting, for each model, the number of target series for which it delivers the best performance under RMSE, MAE, and DA, across the full sample and the pre- and post-COVID subsamples. The relative counts in Table (ref) are sensitive to the set of competing models. In particular, excluding XGB from the comparison increases the number of target series for which MPTE achieves the lowest RMSE, consistent with XGB being a particularly strong benchmark in this empirical setting.Figure (ref) illustrates selected out-of-sample forecasts for GDPC1 and OUTNFB, showing how MPTE and competing models track the realized series over the pre-COVID window and the full evaluation period.
Over the full evaluation period, MPTE exhibits strong RMSE performance. As shown in Table (ref), MPTE delivers the lowest RMSE for several real activity and price-related series, including OUTNFB, PCECTPI, PCEPILFE, CPIAUCSL, and GDPC1. The relative RMSE ranking is broadly stable across subsamples, although the magnitude varies between the pre- and post-COVID periods.
Table (ref) shows that MPTE does not uniformly outperform competing models across all target series. In particular, for several income and trade-related variables, such as DPIC96, EXPGSC1, FPIx, and IMPGSC1, MIDAS or tree-based methods achieve lower RMSE. These series tend to exhibit relatively smooth quarterly dynamics and limited incremental variation at the monthly frequency, which is consistent with more limited gains from explicitly modeling mixed-frequency interactions.
Differences across models are more nuanced when considering DA. As summarized in Table (ref), MIDAS attains the highest DA for a larger number of target series, particularly in the Post-COVID subsample. However, Tables (ref) and (ref) indicate that DA differences between MIDAS and MPTE are often modest at the series level, even when MIDAS ranks first. This pattern suggests that both approaches capture directional movements reasonably well, with MPTE’s advantages emerging more clearly in level forecast accuracy as measured by RMSE rather than sign prediction.
Finally, comparisons with models that incorporate monthly variables only through their end-of-quarter values, rather than modeling within-quarter dynamics, such as OLS, XGB, and NN, indicate that strong forecasting performance can often be achieved without explicitly modeling mixed-frequency inputs. In particular, XGB performs competitively across several target series and subsamples. At the same time, MPTE remains competitive in terms of RMSE across evaluation periods and achieves superior performance for a subset of targets, suggesting that explicitly incorporating high-frequency information can be beneficial when such signals provide incremental predictive content. Overall, the results indicate that the gains from mixed-frequency modeling are context dependent and vary across target series and data regimes.
\FloatBarrier
This subsection examines the empirical relevance of the main architectural components of MPTE by analyzing the same set of ablation variants introduced in Section (ref). Tables (ref) and (ref) report out-of-sample forecasting performance for the same target series as in Tables (ref) and (ref), with target series grouped according to whether the full MPTE achieves the lowest RMSE over the full evaluation period in the comparison with competing models reported earlier. We also report the statistical comparison results in Appendix (ref).
Across target series for which MPTE achieves the lowest RMSE relative to competing models, the impact of removing individual model components varies substantially across targets. For OUTNFB, the full MPTE specification consistently outperforms all ablation variants in terms of RMSE across subsamples, consistent with joint modeling of nonlinearities, attention, and mixed-frequency inputs being particularly valuable for this series.
In contrast, for GDPC1, most ablation variants achieve comparable or lower RMSE than the full model, with AB5, corresponding to a specification that retains only end-of-quarter monthly observations, performing best over the full evaluation period. This pattern suggests that high-frequency inputs provide limited incremental predictive content for this target.
For price-related series such as PCECTPI and CPIAUCSL, no single specification uniformly dominates across all ablations. However, models that retain nonlinear components tend to outperform their linear counterparts, whereas removing attention mechanisms has a comparatively smaller effect on RMSE. This evidence indicates that modest nonlinear transformations can be beneficial for forecasting smoother macroeconomic series, whereas attention-based cross-frequency aggregation appears less critical in these settings. Overall, the ablation results highlight that the relevance of individual model components is target specific and depends on the underlying data characteristics, rather than yielding uniform gains across series.
Table (ref) reports ablation results for target series where MPTE does not achieve the lowest RMSE relative to competing models. Also in these cases, the relative performance of MPTE and its ablation variants varies across targets, and no single architectural component systematically accounts for underperformance.
For some series, such as EXPGSC1, the full MPTE specification outperforms all ablation variants, indicating that removing individual components leads to a deterioration in RMSE performance. In contrast, for targets such as FPIx, specific ablations perform comparably to or slightly better than the full model; in particular, the variant without nonlinear components attains marginally lower RMSE, consistent with limited gains from additional model expressiveness.
For CPILFESL, performance differences across ablations are negligible, with RMSE values differing only at the level of numerical rounding. This near invariance across specifications is consistent with the pronounced smoothness of the series, for which nonlinearities and attention mechanisms appear to provide little incremental benefit.
Overall, the empirical ablation results indicate that the relevance of individual architectural components is highly target dependent and closely related to the amount and structure of information present in each series. In settings where mixed-frequency or nonlinear signals are informative, removing components such as nonlinear transformations or attention tends to degrade forecasting performance, while for smoother series the effects of additional model expressiveness are limited. Motivated by this heterogeneity, we next examine the internal behavior of MPTE by analyzing its attention weights and cross-frequency aggregation patterns, with the goal of better understanding how the model allocates information across variables and time.
\FloatBarrier
\FloatBarrier
Motivated by the target-specific nature of the empirical ablation results, we examine how MPTE allocates information internally across variables and time. We study the learned attention weights to assess whether differences in predictive performance are associated with systematic differences in cross-sectional and temporal aggregation.
A key advantage of MPTE is interpretability through its learned attention weights. To interpret the attention heatmaps, it is useful to distinguish two complementary summaries of the attention matrix. Averaging attention weights over variables yields a distribution over time indices within the context window, which provides an empirical analogue of the temporal weighting operator $B$ studied in the theoretical analysis. Conversely, averaging attention weights over time yields variable-specific importance weights, corresponding to the cross-sectional aggregation induced by the matrix $A_z$. In practice, we compute these summaries by averaging attention weights over all input sequences in the out-of-sample evaluation period. When the optimized model for a given target includes multiple attention heads, we further average the attention weights across heads.
Figures (ref) and (ref) report heatmaps of the cross-sectional attention matrix $A_z$ for selected target series. For each target, we compare the full MPTE specification with the AB1 ablation, which removes nonlinear transformations from the encoder while preserving the attention mechanism. The horizontal axis indexes attending variables, while the vertical axis indexes attended variables.
Figure (ref) shows that introducing nonlinear transformations substantially alters the cross-sectional aggregation structure learned by the model for targets where MPTE achieves lower RMSE than competing approaches. For GDPC1, the full model places pronounced attention weight on variables such as CUMFNS, while the AB1 specification shifts attention toward indicators such as M2REAL and TB6MS. This reallocation is economically intuitive. When nonlinear transformations are available, the model can better capture state-dependent effects in real activity indicators, making capacity utilization in manufacturing (CUMFNS) particularly informative for forecasting GDP growth. In the absence of nonlinearities, such nonlinear effects cannot be captured effectively, and the model instead places greater weight on variables such as the real M2 money stock (M2REAL) and short-term interest rates (TB6MS), whose influence on output is more readily approximated by linear relationships.
A similar pattern emerges for OUTNFB. In both the full model and the AB1 ablation, several quarterly variables receive high attention weight, reflecting a pronounced low-frequency component of nonfarm business output. However, the presence of nonlinear transformations modifies how monthly and quarterly information is combined. Under the full specification, variables such as CPILFESL and IMPGSC1 receive greater attention, consistent with nonlinear interactions between prices and trade-related quantities that provide incremental information for forecasting real activity. In contrast, without nonlinear transformations, the model concentrates attention on income-related aggregates such as GPDIC1 and DPIC96, alongside CPILFESL, consistent with a more linear cross-sectional aggregation strategy.
The reallocation of attention toward capacity utilization when nonlinear transformations are introduced is consistent with a large macroeconomic literature emphasizing state-dependent real activity dynamics. Measures such as manufacturing capacity utilization are known to exhibit regime-dependent and threshold-type effects, becoming particularly informative during expansions and near capacity constraints, while appearing weakly predictive when averaged linearly across business-cycle states (e.g., hamilton1989new,stock1999forecasting,stock2003forecasting). In contrast, monetary aggregates and short-term interest rates summarize average financial conditions and are more readily approximated by linear relationships, explaining their prominence in the ablation without nonlinear features. Similar nonlinear aggregation mechanisms have been documented for low-frequency output measures, where interactions between prices and trade-related quantities provide incremental forecasting power only when nonlinearities are allowed (giannone2008nowcasting; kilian2009not).
These results are compatible with MPTE delivering strong predictive performance relative to competing models, as in the cases of GDPC1 and OUTNFB, the estimated $A_z$ matrix exhibits structured and interpretable patterns rather than diffuse attention. While the specific variables emphasized differ across ablations, attention remains concentrated on economically meaningful subsets of predictors, suggesting that the model is exploiting cross-sectional structure in a systematic manner.
In contrast, Figure (ref) shows that for CPILFESL, a target for which MPTE does not consistently achieve lower RMSE than competing approaches, the cross-sectional attention patterns are substantially less structured. Both the full model and the AB1 ablation distribute attention more broadly across variables, with no clearly dominant predictors emerging. This behavior is consistent with the smooth dynamics of core inflation measures, where incremental nonlinear or mixed-frequency information is limited. In such settings, the model appears unable to extract a clear cross-sectional signal, which is associated with more diffuse attention weights and correspondingly weaker gains from additional architectural complexity.
Overall, the $A_z$ heatmaps are aligned with the empirical findings from the forecasting and ablation exercises. When meaningful nonlinear or mixed-frequency relationships are present in the data, MPTE learns structured cross-sectional aggregation patterns that differ systematically from those of simpler specifications. When such relationships are weak or absent, attention becomes more dispersed, reflecting the limited scope for informative cross-sectional reweighting.
\FloatBarrier
\FloatBarrier
We next examine the temporal attention matrices $B$, which govern how the model aggregates information across time within the input sequence. Figure (ref) reports the estimated $B$ matrices for GDPC1 and OUTNFB for the full MPTE and the AB1 ablation, which removes nonlinear transformations in the encoder. Each matrix summarizes attention weights across a two-year context window, corresponding to 24 monthly lags, where attention flows from earlier to later time steps. Across both targets, temporal attention exhibits clear target-specific structure, and the presence of nonlinearities substantially alters how information from different lags is weighted. For GDPC1, the full MPTE assigns non-negligible attention to older lags, indicating that longer-horizon information contributes to forecasting output growth when nonlinear transformations are available. In contrast, under the AB1 ablation, attention concentrates primarily on the most recent one or two lags, suggesting that in the absence of nonlinear feature extraction the model relies more heavily on short-horizon dynamics.
A similar, though less pronounced, pattern emerges for OUTNFB. When nonlinearities are present, the temporal attention matrix displays a recurring structure consistent with quarterly spacing in the lag index, with relatively higher weights assigned to lags corresponding to multiples of three months. Moreover, the influence of these lags appears to increase with distance into the past, indicating that medium- to longer-horizon dynamics play a role in forecasting nonfarm business output. When nonlinearities are removed, this structure becomes weaker, and attention shifts toward more recent observations. The broader temporal support under the full MPTE specification is economically intuitive. Many macroeconomic forces relevant for output growth, such as capacity constraints, investment cycles, and pricing pressures, evolve gradually and affect outcomes only once certain thresholds are reached. Nonlinear transformations allow the model to convert these slowly moving signals into predictive states, thereby increasing the relevance of longer-horizon information. In contrast, linear encoders are unable to capture such state dependence and therefore rely more heavily on short-horizon dynamics.
Figure (ref) illustrates the effect of removing temporal encoding altogether, corresponding to the AB4 ablation. In this case, temporal attention becomes substantially more diffuse for both GDPC1 and OUTNFB. The model assigns weights with smaller, more uniform magnitudes across lags, with little discrimination among time positions. This pattern reflects the absence of explicit temporal information, which limits the model’s ability to differentiate the relevance of past observations based on their position within the sequence.
Taken together, these results indicate that MPTE endogenously adapts the effective memory of the forecasting model in a target-specific and economically meaningful manner. When nonlinear transformations are available, the model assigns nontrivial weight to medium- and longer-horizon lags, consistent with the presence of slowly evolving macroeconomic forces that influence output growth only once certain thresholds are reached. In contrast, when nonlinear feature extraction is removed, attention collapses toward the most recent observations, reflecting reliance on short-run dynamics that are more readily captured by linear representations. Explicit temporal encoding further sharpens this structure by allowing the model to discriminate among lag positions within the input window. When it is removed, temporal attention becomes diffuse and largely uninformative. Overall, these patterns suggest that nonlinear transformations and temporal encoding jointly enable MPTE to exploit structured temporal dependencies in a disciplined way, rather than mechanically emphasizing either short- or long-horizon information. Unlike MIDAS regressions, which either impose parametric lag polynomials or require selecting a finite lag truncation ex ante, MPTE learns both the relevant temporal horizon and the weighting of individual lags endogenously from the data. This flexibility allows the effective memory length to vary across targets and model specifications.
\FloatBarrier
\FloatBarrier
\FloatBarrier
\FloatBarrier
This paper introduces MPTE, a novel framework that extends classical factor models to handle mixed-frequency data and nonlinear signals through attention mechanisms. Our approach addresses fundamental limitations of traditional methods that require homogeneous sampling frequencies and rely on linear signal extraction, making it particularly relevant for modern high-dimensional economic and financial datasets.
Our main theoretical contribution establishes consistency and asymptotic normality of factor, loading, and common-component estimators in the linear case under general cross-sectional and temporal attention. We show that Target PCA pelger2024target_pca arises as a special case of our framework when attention reduces to fixed block-level projections, while the general attention-based formulation accommodates rich forms of heterogeneity across units, time periods, and datasets. Unlike target-focused PCA methods that impose uniform weighting within blocks, our approach allows the data to determine which cross-sectional units and time periods are informative for the target task. This adaptivity yields an effective sample size that reflects relevance rather than raw dimension and enables genuine transfer-learning gains: auxiliary data can improve estimation precision at the first order without altering identification of the target factor space. The resulting asymptotic theory clarifies how factor and loading uncertainty interact across different regimes and shows that efficiency gains are endogenously determined by the concentration of attention. Conceptually, the attention mechanism plays a role analogous to adaptive shrinkage or thresholding in high-dimensional statistics, replacing fixed linear combinations with data-driven weights that focus on informative observations while down-weighting irrelevant or noisy ones. As a result, our framework strictly generalizes existing Target PCA approaches and is particularly well suited for forecasting environments characterized by heterogeneous relevance, structured noise, and partial factor overlap across datasets.
The empirical applications demonstrate the practical value of our approach. In controlled simulations, MPTE exhibits superior performance in nonlinear environments while remaining competitive in linear settings. The macroeconomic forecasting exercise using 48 mixed-frequency series across 13 target variables shows that our method achieves competitive performance against established benchmarks, with the model remaining robust across different evaluation periods and metrics.
Beyond forecasting accuracy, MPTE offers interpretability through attention weights that reveal target-specific variable importance and effective memory length. By separately aggregating attention across variables and time, we show that the model learns economically meaningful cross-sectional and temporal dependencies endogenously, rather than imposing fixed lag structures. Nonlinear feature extraction shifts attention toward state-dependent indicators and medium- to long-horizon dynamics, while simpler specifications emphasize short-run fluctuations and diffuse signals. This provides a transparent link between predictive gains and their underlying economic sources, offering a data-driven framework for policy-relevant interpretation.
Several avenues for future research emerge from this work. First, extending the theoretical analysis to cover nonlinear activation functions would provide formal guarantees for the full MPTE framework. Second, while the attention mechanisms studied here are adaptive in the sense that they learn heterogeneous relevance across units and time within a stable environment, allowing them to evolve endogenously in response to structural breaks or regime changes would further improve performance in nonstationary economic settings. Third, instead of producing point forecasts, the model could be extended using probabilistic deep learning methods to learn the full predictive distribution of the target variable and provide direct uncertainty quantification. The framework also opens possibilities for applications beyond macroeconomic forecasting. Financial risk management, climate modeling, and energy forecasting all involve mixed-frequency data where our approach could prove valuable.
\cleardoublepage