Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
39,579 characters · 13 sections · 24 citation commands
\runningtitle{Life Sequence Transformer}
\twocolumn[
\aistatstitle{Life Sequence Transformer: Generative Modelling of Socio-Economic Trajectories from Administrative Data}
\aistatsauthor{ Alberto Cabezas \And Carlotta Montorsi}
\aistatsaddress{ Department of Economics and Statistics \\ University of Turin \And Luxembourg Institute of \\ Socio-Economic Research } ]
In fields such as physics geneva2022transformers, chemistry kim2021generative, and health yang2023transformehr, researchers successfully use Transformer-based generative models to simulate surrogate dynamical systems under varying conditions and estimate event probabilities. In addition to their predictive capabilities, generative models have the potential to explore hypothetical system evolutions under different scenarios.
We extend this line of research to the social sciences by proposing a Transformer-based generative model for sequences of human life events. Longitudinal administrative records contain rich information on individuals’ trajectories across labour markets, health, education, and other socio-economic domains, yet these data are rarely treated as sequences for generative modelling. Through a novel encoding method of socio-economic events, we enable the training of a decoder-only Transformer model capable of generating plausible synthetic life trajectories.
Our model is trained on 1.6 million individual labour market histories from the Italian social security administration (INPS), described in Section (ref). The representation of these histories as overlapping socio-economic events encoded into life sequences is introduced in Section (ref). The architecture and training procedure of our life sequence Transformer are detailed in Section (ref). The empirical analysis in Section (ref) illustrates the generative potential of the framework through two benchmarking exercises: the accuracy of simulated sequences in capturing established causal nexus among life events (Section (ref)) and the predictive accuracy on out-of-sample trajectories (Section (ref)).
Once trained, the model can generate realistic continuations of partially input sequences and simulate alternative life paths. In other words, it can create counterfactuals, i.e., plausible “what if” scenarios that describe how events might have unfolded under different circumstances. This ability provides a scalable framework for constructing coherent counterfactual trajectories without the need for external control groups. In Section (ref), we elaborate on how this approach can be integrated into experimental designs in the social sciences and outline promising directions for future research that build on our framework.
Our work builds on recent advances in Transformer-based sequence modelling for longitudinal labour and health data. In particularly, Life2Vec savcisens2024using, CAREER vafa2024careerfoundationmodellabor, and TransformEHR yang2023transformehr. Life2Vec tests the validity of encoder-only Transformer models for modelling socio-economic administrative data from Denmark. It outperforms the predictive performance of standard models, such as mortality tables, logistic regression, or recurrent neural networks, in predicting an individual's future life outcomes (e.g., early mortality) or time-invariant characteristics (e.g., personality nuances). However, this model lacks generative capability to simulate sequences of future tokens, and the data encoding cannot be easily adapted to a generative task. In contrast, our proposed encoding approach enables the training of a generative decoder, allowing for the simulation of counterfactual life sequences on a model architecture directly inspired by Life2Vec.
The CAREER encoder-decoder Transformer model focuses on job sequence generation with annual temporal resolution and no rich occupational attributes, such as firm size, social security benefits, or multiple job episodes within the same year. We enhance this framework by introducing fine-grained temporal encoding and a richer set of characteristics that span labour and demographic dimensions. In CAREER, time is introduced with a timestep of one year, and there is no way to model the duration of events or the exact timing of their occurrence. You could increase the timestep to monthly steps, but this would restrict generation to a fixed monthly grid, offering no control over when events begin or how long they last within the year. Our method of encoding time, by contrast, enables flexible temporal generation by allowing each life event to have a defined starting month and duration in months within the year.
TransformEHR is an encoder-decoder Transformer generative model that encodes sequences of electronic health records and predicts the subsequent closest diagnoses (ICD-10 codes). It includes demographic and time embeddings to model the numerical format of the visit date. TransformEHR is pre-trained to predict a patient's future visits with ICD codes, given longitudinal codes up to the current visit. Similarly to Life2Vec, the timing of each visit must be specified at inference time, meaning the model cannot decide when events occur, but only predict diagnoses for predefined dates. In contrast, our model allows the generation process to determine when events occur and their respective durations. Crucially, our model supports variable event durations, which is essential for modelling life sequences that extend beyond instantaneous diagnoses.
Our model is trained on the Work History Italian Panel (WHIP), a 1:15 sample of individuals drawn from the INPS database leombruni2010note, bena2012new. INPS records cover most private-sector employees, quasi-dependent and self-employed workers, as well as periods of retirement and access to social benefits (e.g., unemployment or maternity leave). The WHIP provides detailed longitudinal employment and benefit histories for over 4 million individuals.
We use the WHIP database for the period 1990–2015 and apply sample selection criteria to obtain sequences that are sufficiently long to provide meaningful information. Specifically, we retain only individuals with complete information up to 2015, at least half of their potential labour force participation period observed, non-missing birth dates, at least five records in total, plausible entry ages (between 15 and 80), and a labour force attachment of at least 10%. This resulted in a sample of around 1.65 million life trajectories. A full description of the dataset and details of the selection procedure are provided in Appendix (ref).
Each labour event contains timing (start month and duration) and income, which is standardised into monthly real terms and discretised into 100 quantiles. Yearly income is the principal variable for the social security system, as it forms the basis for calculating contributions and benefits. Each income record is linked to a range of demographic, workplace, and labour-force-specific attributes. Demographic information includes sex, birth year, and month, and the macro-area of birth. Workplace characteristics cover the sector of activity, enterprise size, and workplace location at the provincial level.
Labour force information is recorded in terms of labour status (e.g., employee, self-employed, pensioner, unemployed) with additional modifiers depending on status. For employees, these include job title, work intensity, number of weeks of maternity leave, number of weeks of sick leave, and whether the contract is part-time or full-time. Together, these attributes enable a detailed reconstruction of individuals’ trajectories across work, unemployment, pensions, and benefit receipt. A full description of all 15 variables and encoding details is provided in Appendix (ref).
We build on the work of savcisens2024using, where life events are concatenated in a sequential structure, with absolute and relative temporal information embedded as auxiliary attributes rather than forming the structural backbone of the sequence. Instead, our encoding method imposes an explicit calendar-like structure on each life trajectory to support the sequential generation of future tokens. The entire observed work history is divided into yearly segments, each further subdivided into twelve monthly intervals, which are then populated with event information as needed. Life events are represented as discrete units with attributes such as employment type, income quantiles, work intensity, sector, qualification, and duration, forming a sequence of socio-economic states.
For instance, if an individual remains employed at the same firm for 10 uninterrupted years, we encode this as 10 events, each lasting 12 months, described using the same set of descriptive tokens, and with a likely increase in income or a change in work title if promoted over the years. If a person holds a one-year contract but experiences a one-month interruption, the year is split into two separate events with different durations. Though the same set of concept-tokens is otherwise used to describe these events. When no event occurs within a year, the structure still records an “empty” year preserving the regular yearly pace of the sequence.
Figure (ref) illustrates the encoding pipeline. We consider each tabular record (i.e., each row in the administrative dataset) as an event and translate each column describing it into a concept-token. Then, we concatenate all the events for a given individual in chronological order and create an individual sequence.
Each event includes embeddings on positional information: year or absolute position reporting the calendar year when the event occurred, age or relative position with the individual’s age at the time of the event, and the usual positional embedding used in Transformer models for the position of the token in the sequence. Notice, these three embeddings are not part of the generative task; instead, they are deterministic.
Sequences start with the special token <BOL> (Beginning of Life). Individual events are then encoded using a fixed token order. <MONTH_X> marks the calendar month the event begins and helps separate multiple events within the same year. Next comes <TYPE_X> to indicate the event type (e.g., employment, unemployment), optionally followed by modifiers like <ATECO_X> (for sector) and <WRK_TITLE_X> (for job title), depending on the event. <DUR_X> defines the event’s duration in months and concludes the block describing the event. If more events follow in the same year, another \texttt{<MONTH_X>} appears; otherwise, the \texttt{<EOY>} (End of Year) token concludes the description of the events in a year. If no further events ensue, the entire life sequence ends with an \texttt{<EOL>} (End of Life) token. Each sequence maintains a consistent ordering of these concept-tokens, ensuring a structured representation and facilitating the learning of this artificial language.
While the experiments in this paper focus on labour-related life sequences, we emphasise that the proposed encoding scheme applies to other life domains, such as health and education. In the health domain, for instance, life sequences could represent concatenations of diagnostic attributes, annotated with the month of diagnosis and the duration of treatment or hospitalisation. Although a monthly resolution may appear too coarse for modelling certain health events, this limitation is mitigated by including intensity tokens <INTENSITY_X>, which capture the extent of engagement within each event's duration. For a detailed description of the data and a labour life sequence example, see Appendix (ref).
Background time-invariant information, such as gender, place, and date of birth, is also represented using concept-tokens (e.g., <F> or <M> for female and male). This information is placed at the beginning of the sequence, forming the entire input as a single stream of tokens. The encoding distinguishes between static demographic attributes and the temporal sequence of life events, which begins with the <BOL> token.
We propose a decoder-only Transformer architecture, see Figure (ref), trained autoregressively to predict the next token in the sequence, conditioned on past events and background information.
The model follows a Transformer decoder-only design, consisting of three components: an embedding block, stacked Transformer-based decoder layers, and a task-specific output head. We make three key modifications to the standard architecture. First, we replace the conventional attention mechanism with Performer Attention choromanski2022rethinkingattentionperformers, which reduces the number of trainable parameters and improves computational efficiency. Second, we incorporate Rotary Position Embeddings (RoPE) su2023roformerenhancedtransformerrotary, adapted to work with Performer Attention, to better capture temporal patterns in the sequence. Third, to stabilise training, we apply RMS Normalisation zhang2019root before the attention and feedforward layers. The decoder uses causal (or masked) self-attention, ensuring that each token can only attend to earlier positions in the sequence. During training, the model is optimised using a next-token prediction objective, learning to generate coherent event sequences conditioned on prior events and time-invariant variables.
\paragraph{Token embeddings}
The first transformation of the input sequence occurs in the embedding layer, where each token in the sequence $S_r$ is mapped to a continuous vector representation using an embedding matrix $\mathcal{E}_\mathcal{V}: \mathcal{V} \rightarrow \mathbb{R}^d$. Each row of $\mathcal{E}_\mathcal{V}$ corresponds to a unique token from the vocabulary $\mathcal{V}$, and each column represents one of the $d$ embedding hidden dimensions. Following savcisens2024using, we center the embeddings by removing the mean from each matrix row.
\paragraph{Time embeddings} To encode temporal information, i.e., age and calendar year, we use the Time2Vec method kazemi2019time2veclearningvectorrepresentation, which allows the model to capture both linear and periodic aspects of time. This method uses trainable parameters $\omega$ and $\varphi$ to define a flexible sinusoidal function. We construct two separate time embeddings: one for age, denoted by $\mathcal{T}_A: \mathcal{A} \rightarrow \mathbb{R}^d$ with $Z := 4$, and one for year, $\mathcal{T}_Y: \mathcal{Y} \rightarrow \mathbb{R}^d$ with $Z := 2$. These embeddings are computed as:
Initial iterations of the model used standard Time2Vec encoding, i.e., a linear component with trigonometric functions. However, we noticed that it led to poor out-of-sample performance: the model quickly lost coherence and failed to maintain the grammar of the life sequence when extrapolating beyond the training period. We hypothesised that this issue stemmed from the unbounded linear trend in the Time2Vec representation of age and year. Additionally, trigonometric components appeared to mimic linear trends, particularly for the year variable, and also for age.
Hence, we adapt the original Time2Vec by varying the proportion of linear and periodic components for different temporal signals and bounding the linear component for better out-of-sample generalisation (see Section (ref)). For year embeddings, we allocate a larger share of the hidden dimensions to the linear component ($\tanh$) to better capture the short-range time frame, which spans only 25 years. For the Age embedding, which spans 70 years in our dataset, we expand periodicity ($\sin$) to reflect cyclical patterns (e.g., retirement or career phases) that recur across broader age bands.
\paragraph{Positional embeddings} In addition to temporal embeddings, we incorporate standard positional encodings to capture the order of tokens within each sequence. Our approach is adapted to use with local and global heads of Performer attention su2023roformerenhancedtransformerrotary: sinusoidal positional encodings combined with RoPE in the global attention heads. Both positional and time embeddings are added to the token embeddings through a ReZero gate bachlechner2021rezero before being passed to the attention layers.
Our dataset consists of 1,652,877 individual life sequences. We divide them into training (70%), validation (15%), and test (15%) subsets. The models are trained for 15 to 20 epochs with early stopping. Each training batch contains 18 sequences, and gradients are accumulated over five steps before an optimiser step, resulting in an effective batch size of 90.
\paragraph{Optimization}
We employ the AdamW optimiser loshchilov2019decoupledweightdecayregularization, with no weight decay applied to bias terms, embedding weights, or normalisation layers. The learning rate is controlled by the One Cycle policy smith2019super with a warm-up phase covering 30% of the training steps. Gradient norms are clipped at a maximum value of 5 to prevent exploding gradients. To avoid overfitting, a dropout rate of 10% is applied to the attention, feedforward, and embedding layers. Dataloaders and projection matrices of Performer attention are reloaded every epoch to ensure robustness in sampling. We use mixed-precision with bfloat16, which we chose to reduce memory usage and improve training efficiency, considering that RMS Normalisation has been shown to improve numerical stability in lower-precision regimes zhang2019root.
\paragraph{Hyperparameters tuning} We sample hyperparameters using the Tree-Structured Parzen Estimator watanabe2023tree and Optuna optuna_2019. Experiments showed that deeper models performed better. Without normalisation, the model misuses the embedding space, often requiring unusually large embedding dimensions to compensate. With RMS normalisation, this effect was mitigated, although the model benefited from longer warmup phases compared to setups using only ReZero. Increasing depth proved more effective than scaling up feedforward or embedding dimensions, or the number of random features. For local attention, we set the window size to cover approximately three complete events, which empirically balanced context coverage and memory use. Ultimately, we selected hyperparameters that maximized model capacity while fitting within the 16GB of VRAM available on an RTX A4000 GPU: 10 decoder layers, with 8 attention heads, 6 local with a window size of 36 and 2 global with projection of 256 random features; embedding size of 240, and position-wise feed forward size of 960 on all layers. The model has around 10.1 million tunable parameters.
In this section, we present the next-token prediction task metrics and evaluate the model’s ability to forecast out-of-sample for years beyond the training period. We then evaluate performance through a series of causal inference benchmarking exercises by comparing the generated life sequences against well-established empirical causal findings from labour economics. All experiments are run on the test subset of our data and code can be found in \href{https://github.com/albcab/life-sequence-transformer}{github/life-sequence-transformer}.
We begin by evaluating the model on a next-token prediction task over the test subset, which measures how accurately it generates the next element in a life sequence given prior tokens and background information. Training each model took approximately 9 hours per epoch for the training phase and 1.5 hours per epoch for validation, totalling 10.5 hours per epoch. The final model was trained for 20 epochs for a total training time of 210 hours on a single NVIDIA RTX A4000 GPU with 16GB of VRAM.
Table (ref) reports a standard set of classification metrics computed across the entire sequence dimension and vocabulary, excluding padding tokens. Each row corresponds to a different length of observed history provided as input. Specifically, we generate the continuation of a life sequence after observing either zero, one, five, or ten years of past events. The metrics include macro-averaged accuracy, precision, recall, and F1 score, each evaluated using top-1 predictions, i.e., consider only the highest probability to assign the predicted token. We also report the square root cross-entropy loss, i.e., perplexity, a widely used measure in language modelling.
As expected, the more historical context the model observes, the better it can predict future tokens. This is consistent with the intuition that individuals often follow relatively stable life patterns after a certain age, e.g., remain within certain industries over time. Access to a longer sequence of past events enables the model to learn person-specific trajectories and reduces uncertainty about what is likely to happen next.
We evaluate the model’s capacity for generating causal relationships by testing whether it reproduces well-established effects from the labour economics literature and national policy. To do so, we truncate observed individual life sequences at specific cutoff years and condition the model on this truncated history. The model is then tasked with generating the continuation of the sequence forward in time. We only truncate at the level of whole years, so that the conditioning context always consists of a realistic, complete history up to the cutoff point. In each experiment, we choose the cutoff year to coincide with the onset of the event of interest (i.e., unemployment, maternity, or retirement), so that the subsequently generated trajectory can be compared to known empirical effects. Additional details of this procedure are provided in Appendix (ref). All generation experiments are run on batches of 8 sequences. On a GPU, generating the next token takes approximately 0.2 seconds per batch, while on a CPU it takes about 2 seconds.
First, we evaluate whether the model captures the impact of an unemployment benefit policy rule in Italy. Until 2017, Italy's mobility allowance granted income support to workers displaced due to firm closure, with benefit duration varying discontinuously by age (cutoffs at 40 and 49) and by region (South vs. North/Central) brunello1997benefit. For each individual, we truncate the sequence at the year when displacement is first observed and provide this truncated history to the model. The model must then generate forward, including the timing and duration of subsequent unemployment and benefit receipt.
Figure (ref) shows the simulated average duration of mobility allowance receipt as a function of age at displacement. The simulated data captures the overall upward trend in benefit duration with age, suggesting that the model has learned the general relationship between age and benefit duration. However, it falls short of reproducing the sharp discontinuity observed at the threshold, instead smoothing over the abrupt policy-induced change. One likely reason is data scarcity: fewer than 1% of sequences involve a mobility allowance spell (see Figure (ref)), which restricts the amount of policy-relevant signal available for training.
A more formal discontinuity analysis confirms this pattern. Table (ref) reports local average treatment effects (ATEs) for different bandwidths around the 40-year cutoff. At narrow bandwidths (e.g.,\ $\pm 12$ months), the model ATE fails to significantly capture the discontinuity. As the window expands to $\pm 48$ and $\pm 96$ months, however, the model’s ATEs move closer to the empirical benchmark, with discrepancies shrinking to below one month and confidence intervals overlapping zero. At the widest bandwidth ($\pm 144$ months), both the empirical and model estimates are around 8–9 months, with a modest and statistically insignificant difference.
The second exercise examines the motherhood penalty, i.e.,\ the long-term career setbacks women face after having children, often including reduced earnings and labour market detachment casarico2023behind, kleven2019children. Here, we simulate women’s careers from the moment of first maternity. Specifically, we truncate sequences either (i) at the year of the first observed maternity leave (offset--0), or (ii) one year before that event (offset--1). The latter setup allows maternity leave itself to be generated or not, and enables counterfactual analysis by comparing career paths with and without maternity.
Figure (ref) compares the real and simulated income trajectories of mothers following their first childbirth (see a breakdown by sector in Figure (ref)). Synthetic sequences are generated conditional on the real sequence up to the maternity leave year. The model successfully captures the key features of the observed trends, closely replicating the average post-birth income dynamics.
To estimate the treatment effect of childbirth on earnings, we use an event-study design with controls for age and calendar year, as in kleven2019children. Because fertility is endogenous, we construct counterfactual trajectories using both propensity-score matched non-mothers and model-generated no-maternity paths (offset--1 simulations). The details of this methodology are presented in Appendix (ref). These event-study coefficients are then summarised into average percentage income penalties over post-birth horizons, making them directly comparable to the ATEs reported for pension and unemployment.
The results show that the model reproduces the empirical child penalties closely across all horizons (Table (ref)). What is especially interesting is the pattern of differences: the model tends to understate the immediate short-run penalty but generates a more persistent effect in the medium and long run. In other words, the simulated trajectories suggest a smaller initial income drop but slower recovery, implying that the penalty lingers for longer.
Figure (ref) illustrates how the model-generated simulated controls (non-maternity paths) compare to real mothers’ earnings trajectories, with propensity score–matched mothers included as a comparison. The simulated controls track the counterfactual trend, reinforcing the finding that the model not only captures the observed income losses of mothers but also produces realistic no-maternity baselines.
The final exercise investigates whether the model replicates the well-documented effect of birth month on retirement timing ardito2020work. Institutional features, such as school starting dates and labour market entry, generate systematic differences in retirement timing between individuals born in January versus those born in December. To test this, we truncate observed sequences prior to retirement. In one variant, we cut off at the year immediately before the observed retirement (so the model must decide whether retirement occurs in the following year or later). In another variant, we truncate four years before retirement, which increases the uncertainty in retirement timing and allows us to compare under different predictive horizons.
Figure (ref) shows results with the one-year cutoff. Here, the model reproduces the negative effect of December birth on retirement age, indicating robustness of the signal. However, a clear upward bias emerges. This suggests that conditioning the model too close to the event may shift its internal expectations and effectively push retirement further into the future.
Figure (ref) reports results with the four-year cutoff. The model captures the expected empirical pattern: December-born individuals retire earlier than January-born peers, and the magnitude of this birth-month effect is well preserved across cohorts. The upward bias in retirement ages is largely absent under this earlier cutoff, indicating that conditioning further from the retirement event leads to a closer alignment with the observed timing of pension entry.
Table (ref) formalises these findings. Both cutoffs generate significant differences in retirement age between treated (born in January) and untreated (born in December) individuals, consistent with the empirical benchmark. Yet the size of the model's effect is sensitive to the conditioning horizon. Truncating the sequence further from the outcome offers a stronger test of counterfactual reasoning. Still, it tends to overstate effect sizes, illustrating how the length of the conditioning horizon shapes both predictive accuracy and the model’s interpretation of the causal nexus among life events.
We evaluate the model’s out-of-sample performance by investigating the coherence of generated life sequences beyond the temporal window of the training data, i.e., after 2015. In this setting, the model is conditioned on observed sequences up to 2015 and generates tokens into the future. Good out-of-sample performance indicates robustness, a desirable property for a model expected to simulate plausible scenarios.
Given the absence of ground truth data for these later years, our evaluation focuses on syntactic and structural correctness rather than predictive accuracy. Specifically, we examine the generated sequences for internal consistency, valid token ordering, and the absence of structural violations, such as overlapping events or misaligned episode boundaries (more details in Appendix (ref)). This assessment allows us to verify that the model preserves the learned “grammar” of life sequences when extrapolating beyond the observed data.
Figure (ref) presents the density of out-of-sample generated years in which the first structural failure occurs, considering only sequences where a failure did occur within a 20-year generation window. The results show that sequence generation begins to fail around year 7 or 8, though a non-negligible share persists further, with successes becoming exponentially less likely as the generation horizon extends, and 5% of sequences remain structurally valid after 20 years.
We propose a novel encoding method for converting longitudinal administrative data into sequences and train a generative life sequence Transformer model. Despite its relatively small size, the model learns to reproduce realistic and causally consistent labour market histories and generates coherent synthetic life paths conditional on past trajectories, showing the potential of Transformer architectures in this domain.
Through empirical exercises, we study the strengths and limitations of our model. On the one hand, it successfully reproduces causal patterns, from birth-month effects on retirement to the motherhood penalty. On the other hand, the exercises reveal systematic sensitivities. Rule-driven discontinuities are smoothed over when the underlying events are rare in the data; treatment effects vary depending on how counterfactuals are defined, whether by comparing mothers to non-mothers or by truncating retirement histories at different horizons; and the timing of effects can shift, with the model sometimes understating short-run penalties but prolonging them over the longer term.
These findings highlight a broader challenge: counterfactual inference with generative models depends on how we define treatment, select controls, and decide where to cut the input history. These are not just technical choices but conceptual ones, raising fundamental questions about what constitutes a life sequence counterfactual. They also suggest future research on developing principled methods for introducing researcher-imposed variation into simulated life sequences, ensuring theory-guided design of counterfactual scenarios.
Finally, while our results show encouraging alignment with empirical findings, they also remind us that predictive modelling of individual life courses carries ethical and societal risks. Structural biases present in training data will be reinforced. Without appropriate human oversight, there is also the potential for misuse, for example, in automated decision-making systems that influence resource allocation or eligibility for benefits. Future applications must combine methodological advances with safeguards for fairness, privacy, and responsible use.
The authors acknowledge European NextGenerationEU funding through the PNRR project “Age-It - Ageing well in an ageing society”, code PE0000015, MUR 341/2022 tender.