Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
50,078 characters · 31 sections · 47 citation commands
AI-Assisted Economic Measurement from Survey Instruments: Evidence from Public-Employee Pension Choice
Survey instruments remain central to economic measurement. They operationalize latent constructs, including beliefs, expectations, constraints, preferences, and knowledge, through structured questionnaires. By compressing complex economic concepts into concise item stems, researchers can field large-scale instruments and collect rich data on individual behavior and outcomes. This compression, however, creates a fundamental measurement challenge. Many survey items bundle multiple mechanisms within a single question, so any strict one-to-one assignment of items to constructs can be problematic. When an item reflects both beliefs and knowledge, or both constraints and preferences, the resulting measurement scores can blur conceptual boundaries and mix distinct economic mechanisms. We call this phenomenon construct contamination, and it has important consequences for empirical analysis. Contaminated constructs are harder to interpret, less portable across contexts, and they complicate the design of targeted policy interventions cronbach1955construct, campbell1959convergent.
Recent developments in large language models openai2023gpt, bommasani2021opportunities create new opportunities to address this measurement challenge. These models can parse item semantics systematically at scale and produce replicable, text-based classifications of survey content. A growing literature in psychology and measurement science shows that embeddings and LLMs can quantify semantic overlap across items, diagnose taxonomic inconsistencies, and recover questionnaire structure from text alone boke2025observer, huang2025embedding, kambeitz2025empirical, wulff2025semantic, hommel2025language. This work provides a methodological foundation that we build upon. The language of a survey instrument contains actionable information about its measurement structure, and LLMs offer scalable tools to extract that structure in a transparent and auditable manner.
For economics, the central question is not whether LLMs can classify survey items, but whether such classifications improve economic measurement in ways that matter for empirical analysis and policy. For an LLM-based taxonomy to be useful, it must deliver clearer separation of economic mechanisms and more informative heterogeneity in outcomes, and these improvements must be stable out of sample athey2019machine, mullainathan2017machine. Simply expanding the number of constructs can mechanically improve in-sample fit without enhancing measurement quality, and LLM-generated mappings can introduce new forms of overlap and contamination. A credible framework for AI-assisted economic measurement therefore requires coupling scalable measurement construction with rigorous econometric validation chernozhukov2018double. We evaluate the value of an LLM-proposed taxonomy by the same standards applied to any measurement system. Does it improve our ability to predict outcomes, does it sharpen our understanding of mechanisms, and do these gains replicate beyond the sample used to construct the measures kleinberg2015prediction?
We propose an iterative measurement framework that treats the survey instrument itself as data and uses econometric diagnostics to discipline LLM-generated measurement proposals. Starting from item stems, an LLM induces a multidimensional taxonomy of subdimensions and produces sparse item-to-subdimension weight vectors, adapting methods from kargupta2025taxoadapt. We interpret these weights as a soft mapping in which each item loads on a simplex over subdimensions, which makes cross-loading explicit and auditable. We then combine this mapping with harmonized survey responses to construct respondent-level scores for each subdimension. Critically, we do not treat the LLM output as ground truth. Instead, we evaluate each proposed construct using incremental out-of-sample validity tests and discriminant validity diagnostics designed to detect construct overlap borsboom2004concept. When diagnostics indicate contamination or instability, we apply targeted refinement operators that modify the taxonomy or tighten mapping constraints. We iterate until overlap diagnostics stabilize and out-of-sample gains plateau. In this way, the framework operationalizes a discipline mechanism in which added measurement flexibility is retained only when justified by stable empirical performance.
We illustrate the framework using a large-scale survey of public employees on their valuation of defined benefit pension plans relative to defined contribution alternatives, originally analyzed by giesecke2022much. Our out-of-sample diagnostics show that only a small subset of measurement subdimensions delivers stable incremental information for predicting retirement plan choices. The strongest signal is concentrated in mechanisms related to tenure and career-stage lock-in, which improves prediction for both acceptance of a hypothetical plan switch and the required employer contribution rate. Financial literacy content contributes meaningfully to predicting acceptance but provides minimal incremental value for predicting required contribution rates, a pattern consistent with the original study's findings but now validated through systematic out-of-sample testing. The diagnostics also identify construct overlap between literacy and perceived plan generosity, which motivates a targeted refinement that separates beliefs about plan value from objective financial knowledge. This refinement improves discriminant validity without inflating the taxonomy unnecessarily. Overall, the exercise shows how our framework can convert a high-dimensional survey instrument into a validated set of economic measurements, clarify which mechanisms matter for specific outcomes, and guide improvements in survey design.
The remainder of the paper proceeds as follows. Sections 2--5 develop a general framework for AI-assisted economic measurement and its validation. Sections 6--7 apply the framework to a large public-employee pension survey. Sections 8--9 discuss implications and conclude.
Our contribution lies at the intersection of economic measurement, machine learning, and text-as-data methods. We situate the framework within three closely related literatures: semantic measurement and text-based approaches, iterative taxonomy construction, and econometric validation of machine learning--assisted measurement. We also place the contribution in the context of the emerging literature on applications of artificial intelligence in economic research
The use of text-based methods in economics has expanded rapidly over the past decade. gentzkow2019text and grimmer2022text provide comprehensive overviews of text-as-data methods in the social sciences, emphasizing the value of treating language as a source of structured information. benoit2020text offers a complementary perspective focused on the methodological challenges of converting text into structured data, while ash2023text survey text algorithms in economics with applications spanning policy analysis, financial markets, and organizational behavior. Representative applications include kelly2021text, who develop methods for text selection in economic prediction, hansen2018transparency, who analyze Federal Reserve communications using computational linguistics, and baker2016measuring, who construct indices of economic policy uncertainty from news text.
Our approach extends this literature by applying semantic analysis not to naturally occurring text corpora, but to the controlled language of survey instruments, where each item is deliberately designed to measure an underlying construct. While existing work demonstrates that item language contains recoverable structure, it does not address how such structure should be disciplined using econometric criteria.
Recent work in psychology and measurement science shows that embeddings and large language models can recover psychometric structure from item text alone. wulff2025semantic demonstrate that semantic embeddings reveal taxonomic inconsistencies in psychological measurement and can guide construct refinement. boke2025observer use LLMs to assess content overlap in mental health questionnaires, providing observer-independent diagnostics of item redundancy. hommel2025language show that language models can infer correlations between psychological items and scales from text alone, while huang2025embedding and kambeitz2025empirical provide additional evidence that LLMs capture meaningful structure in psychological constructs and identify redundancy in existing instruments.
These studies establish that the language of measurement instruments is informative about construct boundaries and empirical relationships. We adopt this premise but impose a stricter evaluation standard. Rather than treating semantic structure as an endpoint, we treat LLM-generated taxonomies as measurement proposals that must be validated through econometric performance. A taxonomy is retained only if it improves out-of-sample prediction and passes diagnostics designed to detect construct contamination. Our framework adapts this logic to survey instruments, replacing corpus-coverage objectives with out-of-sample economic validation.
The iterative refinement component of our framework draws on recent work in natural language processing on adaptive taxonomy construction. kargupta2025taxoadapt develop TaxoAdapt, an LLM-based procedure for constructing and refining multidimensional taxonomies aligned with evolving research corpora through repeated classification, expansion, and consolidation. We adapt this logic to the finite corpus of survey items, treating the instrument itself as the target corpus. Crucially, we replace corpus-coverage criteria with econometric diagnostics to govern iteration.
The key distinction is that we do not expand the taxonomy to maximize semantic coverage. Instead, we refine it to minimize construct overlap while maximizing out-of-sample predictive value. Iteration is therefore disciplined by empirical performance rather than by textual completeness.
Our emphasis on out-of-sample validation connects to a broad literature at the intersection of machine learning and econometrics. mullainathan2017machine and athey2019machine provide foundational frameworks for incorporating machine learning into economic research, emphasizing prediction tasks, cross-validation, and the avoidance of overfitting. abadie2020sampling clarify the distinction between sampling-based and design-based inference in regression analysis, a distinction that is relevant for interpreting out-of-sample performance in survey contexts. kleinberg2015prediction distinguish prediction policy problems, where forecast accuracy is central, from causal inference problems, where estimating treatment effects is the goal.
Our framework addresses a measurement problem that lies between these categories. We use predictive performance as a diagnostic for measurement quality, while the ultimate objective is to construct portable measures of economic mechanisms that can support causal inference and policy analysis. Importantly, we apply these principles to measurement construction itself rather than only to downstream estimation.
The validation protocol we develop draws on established practices in machine learning for model selection and performance evaluation. We use embedded cross-validation to guard against post-selection bias induced by iterative refinement, following principles in chernozhukov2018double and athey2025surrogate. The incremental validity tests we apply are analogous to methods used in psychology for construct validation cronbach1955construct, campbell1959convergent, but they are adapted to an out-of-sample prediction context. By requiring that each proposed construct improve predictive performance in held-out data, we impose a discipline that prevents the taxonomy from expanding to fit noise.
The discriminant validity diagnostics we employ are grounded in classic psychometric theory. campbell1959convergent introduce the multitrait-multimethod matrix as a tool for assessing convergent and discriminant validity, arguing that constructs should correlate with related measures while remaining empirically distinct from conceptually different constructs. belch1977discriminant provide early applications of discriminant validity assessment in consumer research. cronbach1955construct and borsboom2004concept emphasize that construct validity is an ongoing process of empirical testing rather than a one-time certification.
Our framework operationalizes these principles in a machine learning context by treating high correlations among purportedly distinct constructs as diagnostic of contamination. We use conditional contribution tests to determine whether overlap-prone constructs add unique information beyond existing measures.
Our work also contributes to a growing literature on applications of artificial intelligence in economics beyond conventional prediction tasks. varian2018artificial discusses how artificial intelligence is reshaping industrial organization and economic analysis, while einav2014economics examine how big data is transforming empirical economics. brynjolfsson2021productivity document complementarities between general-purpose technologies such as AI and organizational intangibles, highlighting the importance of measurement systems capable of capturing evolving relationships. hartley2024labor study the labor market effects of generative AI, underscoring the need for robust measurement frameworks as AI adoption accelerates.
Related work examines the use of language models in survey and labor market contexts. argyle2023out explore the use of language models to simulate human survey responses, raising questions about validation and bias. horton2017effects show that algorithmic recommendations can affect labor market matching and outcomes. raji2020closing develop frameworks for algorithmic auditing and accountability, emphasizing transparency and validation in deployed AI systems. Our application differs from this literature in that we use LLMs not to replace human respondents or generate recommendations, but to audit and improve measurement systems. The framework provides a disciplined methodology for leveraging AI to enhance data quality and measurement transparency.
Relative to this literature, our contribution is threefold. First, we develop an instrument-level measurement framework that maps survey items to a sparse distribution over subdimensions, making cross-loading explicit and auditable. Because the mapping is defined at the instrument level rather than the sample level, it is portable across datasets that field the same or similar items. Second, we integrate econometric diagnostics directly into the measurement construction process, using incremental out-of-sample performance and stability to determine which constructs to retain, refine, or discard. Third, we operationalize a disciplined refinement loop in which overlap diagnostics trigger targeted taxonomy modifications and constraint tightening, ensuring that added flexibility is used only when it delivers stable empirical gains. The result is a transparent and replicable procedure for converting survey instruments into validated economic measurements.
This section defines the measurement framework abstractly, independent of any particular dataset. We describe the objects, the LLM-based proposal mechanism, and how these elements fit into an iterative measurement loop.
Let survey items (question stems) be indexed by $j=1,\dots,J$ and respondents by $i=1,\dots,N$. Each item $j$ has a fixed stem text $q_j$ (and, where relevant, answer options). Let $x_{ij}\in\mathbb{R}$ denote respondent $i$'s harmonized response to item $j$ (possibly missing). The goal is to measure a set of latent subdimensions indexed by $k=1,\dots,K$ organized in a multidimensional taxonomy.
We construct two objects: an item-to-subdimension weight matrix $W=\{w_{jk}\}\in[0,1]^{J\times K}$ and respondent-level subdimension scores $S_{ik}$ obtained by weighted aggregation of harmonized responses.
Inspired by TaxoAdapt kargupta2025taxoadapt, we use an LLM to propose a data-driven taxonomy from the set of item stems. TaxoAdapt develops an iterative, LLM-based procedure for constructing and refining a multidimensional taxonomy that is aligned to the topical distribution of a target corpus (via repeated classification, expansion, and consolidation). In our setting, the “corpus” is the finite collection of survey question stems (and, where relevant, response-option text).
\paragraph{Step 0: researcher-defined coarse dimensions (anchors).} Before invoking the LLM, the researcher supplies a small set of economically interpretable, coarse dimensions e.g., beliefs and constraints. These anchors serve two purposes: they constrain the space of admissible mechanisms for interpretability, and they reduce the risk that the taxonomy drifts into idiosyncratic semantic distinctions that are hard to map to economic theory.
\paragraph{Step 1: first-layer subdimension proposal within anchors.} Conditional on the anchor dimensions and the full item inventory, the LLM proposes a first-layer set of subdimensions within each anchor. Each subdimension is accompanied by (i) a short, mechanism-oriented definition, (ii) inclusion/exclusion boundary rules, and (iii) a list of representative items that motivated the label. The LLM sees question text only (not outcomes), so taxonomy induction cannot “fit” behavior by construction.
\paragraph{Step 2: consolidation and audit.} We consolidate near-duplicate candidates and impose naming and boundary consistency checks to obtain an initial taxonomy $\mathcal{T}^{(0)}$ with $K$ subdimensions. We treat the resulting taxonomy as provisional: subsequent econometric diagnostics (Section (ref)) determine whether it is retained as-is or refined using the operators in Section (ref). The resulting initial taxonomy, including top-level dimensions, subdimensions, and their motivating survey items, is summarized in Appendix Table (ref).
\paragraph{Reproducibility.} All prompts used for taxonomy induction and mapping are versioned and reported in Appendix (ref), so the taxonomy is auditable and can be regenerated under the same protocol.
Given a taxonomy $\mathcal{T}$, we map each item $j$ to a sparse distribution over subdimensions. For each item stem $q_j$, the LLM outputs weights $\{w_{jk}\}_{k=1}^K$ such that
To reduce noise and enhance interpretability, we restrict mappings to be sparse (e.g., at most $m$ nonzero weights per item), dropping very small weights and renormalizing to satisfy (ref). This soft mapping captures multi-mechanism item content while keeping item representations comparable.
Survey items differ in response type: binary, ordinal, categorical, and numeric fields. To aggregate responses across heterogeneous items, we first map each response into a unified numeric representation $x_{ij}$ on an item-by-item basis.
Operationally, we (i) preserve ordering when meaningful (e.g., Likert scales), (ii) map categorical bins to ordered numeric codes when an ordering is substantively justified, (iii) apply monotone transforms for skewed numeric inputs when needed (e.g., log1p), and (iv) treat non-substantive responses (e.g., “Prefer not to say”) as missing. For comparability across heterogeneous scales, we winsorize and standardize within the training fold and then apply the same transformation to the test fold to avoid leakage in out-of-sample evaluation. Appendix (ref) documents the full item inventory and harmonization rule used for each item.
Given $W$ and harmonized responses $\{x_{ij}\}$, we construct respondent-level subdimension scores by weighted aggregation:
excluding missing $x_{ij}$ from both numerator and denominator.
To benchmark the soft-mapping system, we also construct a baseline measurement system that assigns each item to a single dimension (hard mapping) and aggregates within dimension using standard index construction rules. The baseline serves as a parsimonious comparator.
The key difference is conceptual: the baseline enforces a one-item-one-construct restriction, whereas the LLM mapping permits sparse cross-loading on the simplex. This matters when item wording bundles mechanisms. Accordingly, we interpret LLM weights $W$ as measurement proposals that can be validated.
Sections (ref)--(ref) turn the soft mapping in (ref)--(ref) into a disciplined iteration procedure. The LLM serves as a proposal engine, suggesting candidate constructs and sparse item weights, while econometric diagnostics determine whether these proposals are retained, modified, or discarded based on incremental out-of-sample value and discriminant validity.
This section defines the validation rules that govern any application of the measurement framework. No empirical results are reported here. Instead, we specify how candidate constructs proposed by the LLM are evaluated, refined, or discarded using econometric criteria.
We validate the measurement system along two dimensions. First, we assess whether adding a candidate subdimension improves predictive or explanatory performance out of sample relative to baseline specifications that exclude it. Second, we evaluate whether the candidate is empirically distinct, in the sense that it captures signal not already absorbed by nearby constructs. Together, these checks operationalize an econometrics-as-arbiter principle: the LLM proposes constructs and weights, while data-driven diagnostics determine which proposals are retained and which are revised.
For each candidate subdimension, we compute an incremental out-of-sample (OOS) gain relative to a benchmark specification without the candidate.\footnote{The protocol supports any scalar OOS metric, including $R^2$, RMSE, log loss, or AUC, depending on the downstream task. Appendix (ref) provides implementation details.} We emphasize both magnitude and stability across folds. Components are classified as signal when gains are positive and stable, as weak signal when gains are small but stable, and as noise-like when gains are near zero or unstable across folds.
Out-of-sample performance alone does not distinguish between true null effects and construct contamination. We therefore pair OOS triage with simple and interpretable separation diagnostics designed to detect overlap and misassignment. These include high correlations with nearby subdimensions, concentrated cross-loadings in the item-to-construct mapping, and failure of conditional contribution tests within overlap clusters. Formal definitions and recommended thresholds are provided in Appendix (ref).
We implement a two-stage triage procedure. In Stage 1, each candidate component is labeled according to its OOS performance as signal, weak signal, or noise-like. In Stage 2, noise-like components are diagnosed as either data-limited, reflecting insufficient coverage or support, or overlap-driven, reflecting empirical inseparability from nearby constructs. Actions follow directly:
This subsection defines the refinement operators used when a component is noise-like in incremental OOS tests and diagnostics indicate overlap-driven contamination (e.g., persistent high correlations, concentrated cross-loadings, or failure of conditional contribution within an overlap cluster). Refinement is applied locally: we modify only the affected neighborhood of constructs and items, while holding unrelated dimensions fixed. All updates preserve the simplex constraint in (ref).
\paragraph{Operator 1: Anchoring.} Anchoring fixes a subset of dimensions whose measurement is treated as stable. For anchored dimensions, item-to-dimension weights are held invariant across iterations: \[ w^{(a)}_{j d} \in \{0,1\}, \quad \sum_{d \in \mathcal{D}_a} w^{(a)}_{j d} \le 1. \] Anchors serve as reference standards and prevent global relabeling during refinement.
\paragraph{Operator 2: Local splitting.} When overlap concentrates within a parent subdimension $d$, we introduce a small set of child subdimensions $\{d_1,\ldots,d_K\}$ nested within $d$ and reallocate content locally. Re-estimation is restricted to items in the affected neighborhood $\mathcal{I}(d)$: \[ w_{j d_k} \ge 0,\quad \sum_{k=1}^K w_{j d_k} \le 1, \quad \text{for } j \in \mathcal{I}(d), \] while anchored and unrelated dimensions remain fixed.
\paragraph{Operator 3: Constraint tightening.} When overlap is driven by diffuse cross-loading or many small weights, we tighten mapping constraints while preserving the simplex structure. Examples include enforcing a one-primary-plus-at-most-one-secondary loading rule, thresholding small weights (followed by renormalization), and constrained relabeling for a small set of items identified as overlap drivers.
\paragraph{Stopping rule.} Iteration stops when overlap diagnostics no longer indicate concentrated cross-loadings among retained constructs and incremental OOS gains plateau across successive refinement rounds.
This section describes the survey data and application used to illustrate the measurement framework. The application draws on a large-scale survey of U.S.\ public employees designed to elicit how workers value continued accruals under their incumbent defined benefit (DB) pension plans relative to a hypothetical switch to a defined contribution (DC) alternative.
Our empirical application uses the public-employee retirement plan survey studied by giesecke2022much. The survey is designed to measure how public employees value future DB accruals relative to a forward-looking switch to a DC plan on a hard-freeze basis. Under this scenario, already-accrued DB benefits are preserved, but no additional DB benefits accrue going forward. In a DB plan, the employer guarantees retirement benefit payments and bears investment risk, whereas in a DC plan the employer makes fixed contributions into an individual account and employees choose how to invest.
The valuation task therefore maps into a compensating differential. Respondents are asked whether they would enroll in a hypothetical DC plan that replaces future DB accruals and, conditional on enrollment, what minimum employer contribution rate, expressed as a percentage of payroll, would make them indifferent between remaining in the DB plan and switching to the DC alternative giesecke2022much. These responses provide direct measures of both discrete acceptance and the intensity of valuation.
giesecke2022much field the survey via email to public employees in U.S.\ state and local government, state higher education, and K--12 school districts across 16 states. They compile approximately 396{,}948 publicly available email addresses and obtain 7{,}674 completed responses. After accounting for bounce-backs from inactive or inaccurate addresses, this corresponds to an adjusted response rate of approximately 2.1%. While modest, the sample size is large for a detailed pension valuation survey and supports rich heterogeneity analysis.
Beyond the DB--DC indifference elicitation, the survey collects detailed information on employment and pension status, perceptions and beliefs about the incumbent plan and the DC alternative, financial literacy, and a broad set of demographic and socioeconomic characteristics. Perception and belief measures include assessments of plan generosity and perceived financial stability, which are central to the valuation of retirement benefits.
Financial literacy is measured using a short battery of basic household finance questions. giesecke2022much construct a financial literacy score defined as the percentage of questions answered correctly, ranging from 0 to 100, and refer to this block as questions Q34--Q40 in the survey instrument. In addition to subjective perceptions, the analysis links respondents to objective measures of plan generosity. Specifically, employer service cost as a percentage of payroll is obtained from pension plan financial disclosures and used as a proxy for the actuarial value of DB accruals.
The application focuses on two outcomes that correspond directly to the survey's core elicitation. The first is acceptance of the hypothetical DC option. The second is the minimum required employer contribution rate, expressed as a percentage of payroll, measured among respondents who indicate willingness to switch. The paper documents high acceptance rates overall and summarizes the distribution of required contribution rates among accepters.
In downstream specifications, we follow giesecke2022much in using a comprehensive set of baseline covariates. These include demographics and job characteristics, such as age and years of service; socioeconomic controls, including household income; cognitive proxies, such as financial literacy and educational attainment; and plan-level measures, including employer service cost as a percentage of payroll. The richest specifications include state fixed effects.
This section reports the empirical results from applying the AI-assisted measurement framework to the public-employee retirement plan survey. The workflow links (i) LLM-based soft mapping of survey items into a multidimensional measurement taxonomy, (ii) harmonization and construction of respondent-level measurement scores, and (iii) out-of-sample (OOS) econometric tests that distinguish predictive signal from noise. OOS diagnostics serve both as evaluation tools and as discipline mechanisms for iteration: the taxonomy is refined only when diagnostics indicate systematic construct overlap that can be resolved through a more interpretable decomposition. Supporting mapping artifacts and refinement allocations are reported in the Appendix.
We evaluate each constructed subdimension using incremental out-of-sample validity tests (Section (ref)). For the binary acceptance outcome, we report $\Delta$AUC and $\Delta$LogLoss. For the required employer contribution rate, estimated on the acceptor subsample, we report $\Delta R^2$ and $\Delta$RMSE (RMSE is reported as baseline minus augmented). Components are interpreted jointly by magnitude and stability across folds.
Table (ref) reports full cross-validated incremental validity results for acceptance. Two components stand out. service\ tenure\ lockin delivers the largest and most stable gains ($\Delta$AUC $\approx 0.114$), and financial\ literacy also contributes strongly ($\Delta$AUC $\approx 0.082$). By contrast, employment\ context and the initial perceived_generosity aggregate behave as noise-like.
Table (ref) reports the corresponding results for the required contribution rate among accepters. The dominance of service\ tenure_lockin persists on this intensive margin ($\Delta R^2_{\text{OOS}} \approx 0.0089$), while most other components contribute weakly or not at all.
Incremental OOS performance does not by itself distinguish between true null effects and construct contamination. We therefore complement OOS triage with overlap diagnostics that proxy for discriminant validity.
Table (ref) reports all constructed subdimension pairs with $|\rho|\ge 0.85$. A single, pronounced overlap emerges: financial_literacy and perceived_generosity have correlation 0.948. This near-collinearity is consistent with the cross-loading pattern observed in the item-level mapping and motivates targeted refinement rather than wholesale discarding.
Guided by the overlap diagnostics, we implement a targeted refinement that anchors the canonical Financial Literacy battery (Q34--Q40) and decomposes perceived_generosity into more granular belief channels. The allocation of item weight mass across child constructs is reported in Appendix Table (ref), and the refined OOS incremental validity results are reported in Appendix Table (ref). The refinement prompt and constraints used to implement the split are documented in Appendix Figure (ref).
We assess robustness along three dimensions: sparsification thresholds, alternative score construction rules, and permutation placebo tests. Robustness tables are reported in Appendix (ref). In particular, Appendix Table (ref) shows that overlap diagnostics and incremental OOS performance are stable across sparsification thresholds and top-$m$ constraints, while Appendix Table (ref) reports permutation placebo results that destroy semantic alignment while preserving mechanical complexity.
Figure (ref) summarizes threshold robustness across weight thresholds $\tau$ and top-$m$ constraints. Figure (ref) reports placebo distributions that preserve mechanical complexity while destroying semantic alignment. Robustness tables, including Table (ref) and other grid summaries, are available in Appendix (ref).
For completeness, Appendix Table (ref) consolidates the full out-of-sample triage across all constructed subdimensions, reporting incremental performance, stability classifications, and refinement outcomes. This table summarizes how the diagnostic rules map candidate constructs into retained signal, weak signal, and discarded noise.
The empirical results in Section (ref) illustrate how the framework operates in practice and how it can be used to extract, validate, and refine latent economic mechanisms encoded in survey wording. The application demonstrates how a rich instrument can be converted into a transparent and testable measurement system.
Several patterns emerge consistently from the results. First, economically meaningful signal is highly concentrated. Across both outcomes studied---acceptance of a hypothetical DC option and the minimum required employer contribution rate---the strongest and most stable incremental information is captured by the service_tenure_lockin channel. This dimension survives out-of-sample (OOS) validation across folds and remains robust under refinement, indicating that career-stage incentives and lock-in considerations represent the most reliably measured mechanism in the instrument. Second, financial literacy content contributes meaningfully to predicting acceptance but adds little stable incremental value for the required contribution rate among accepters. Third, several intuitively appealing constructs behave as noise-like when evaluated as aggregate indices, but targeted refinement can recover interpretable weak-signal belief channels, including a plan-stability channel that delivers materially larger incremental gains for acceptance.
These findings align naturally with, and build upon, the empirical patterns documented in giesecke2022much. Their analysis shows strong heterogeneity by years of service and documents that financial literacy is associated with acceptance but largely unrelated to the intensive margin of valuation. Our framework preserves these baseline relationships for comparability, but adds a distinct measurement layer that formalizes them. Rather than relying on pre-specified, hard-coded indices, we allow survey items to load flexibly on multiple latent subdimensions and then use OOS diagnostics to determine which of these dimensions are empirically supported. In this sense, the framework complements existing applied work by converting reduced-form patterns into explicit, validated measurement objects, while avoiding over-interpretation of dimensions that do not survive validation.
From a practical perspective, the framework offers applied researchers a way to work systematically with complex survey instruments. Because the mapping is defined at the level of item text rather than estimation samples, it is portable across survey waves and related instruments. The diagnostics identify which parts of an instrument contain reliable signal for particular outcomes and which domains are too coarse, redundant, or weakly connected to behavior. In the pension setting, the results highlight the central role of lock-in mechanisms for both extensive and intensive margins, while suggesting that informational content is most relevant for take-up decisions rather than valuation. More generally, the approach can guide survey redesign by indicating where additional items are likely to improve measurement and where existing batteries may be redundant.
Several limitations are worth noting. The framework evaluates measurement quality relative to specific outcomes, so constructs that appear noise-like here may be relevant in other applications. Finite instruments constrain the granularity of recoverable mechanisms, and OOS gains are necessarily modest in weak-signal settings. Moreover, while semantic tools reduce the burden of parsing item content, economic judgment remains essential in selecting anchors, interpreting refined constructs, and assessing policy relevance. These limitations are inherent to measurement rather than specific to the use of language models.
Despite these caveats, the results demonstrate that combining semantic analysis with econometric validation can sharpen economic measurement in a transparent and replicable way. By making ambiguity explicit, disciplining flexibility with OOS diagnostics, and anchoring interpretation in established economic theory, the framework provides a general approach for extracting mechanism-relevant information from survey instruments while remaining firmly grounded in empirical economics.
We propose a diagnostic-driven, AI-assisted framework for economic measurement that starts from survey instruments and ends with validated, portable measurement objects. The framework combines simplex-constrained soft mapping from item text to latent subdimensions, unified response harmonization across heterogeneous item formats, and an econometric validation protocol based on out-of-sample incremental performance and overlap diagnostics. Crucially, outputs from large language models are treated as measurement proposals rather than as ground truth, with transparent decision rules determining which constructs are retained, refined, or rejected as noise. Applied to a public-employee retirement plan survey, the framework shows that only a small subset of candidate subdimensions delivers stable incremental information beyond baseline measurement: lock-in mechanisms generate the strongest gains for both acceptance and required contribution rates, while financial literacy contributes meaningfully to acceptance but only weakly to the intensive margin. These patterns align with existing qualitative interpretations in the application setting while adding a validation-based interpretation, showing that weak intensive-margin relationships persist under cross-validation rather than reflecting in-sample specification choices. More broadly, the contribution is methodological. The framework provides a practical way to translate high-dimensional survey instruments into economic measurements that are auditable, reusable, and disciplined by standard econometric criteria rather than by semantic richness or in-sample fit. Future work can extend the validation suite to other outcomes, settings, and survey designs, and examine when soft mappings generalize across instruments that differ in wording, ordering, or response formats.
Disclosure. None of the authors have financial or other ties related to the research project.