Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
33,557 characters · 14 sections · 12 citation commands
Causality Elicitation from Large Language Models
Causal analysis often begins by identifying key events and specifying hypothesized mechanisms that connect them. The advent of large language models (LLMs) enables us to automate document summarization, event extraction, and the generation of causal narratives, thereby enriching such analyses. Moreover, the knowledge encoded in LLMs can be leveraged to propose causal hypotheses. Importantly, our output graph is a “hypothesis map”, not a validated real-world causal model.
This study develops a method to elicit causal relationships among events from LLMs. We view each generated document as a topic-conditioned scenario sample and encode event co-occurrence in an incidence matrix. For that purpose, we propose a general pipeline consisting of five steps:
The motivation is to establish a procedure for extracting causal hypotheses in a sophisticated, interpretable, and robust way within the framework of causal inference.
Steps (iii) and (iv) address the problem of variation in event names. If events extracted by an LLM are treated as free-form text, the “variable identity” problem arises: the same underlying event may appear under multiple names across documents. For example, suppose we obtain the following extractions from a discussion of trade policy:
These statements may essentially refer to the same underlying event, which we might call “tariff tightening.” However, if we treat raw strings as distinct variables, they are counted as separate columns. This not only makes interpretation difficult but also undermines the stability of downstream comparison, feature selection, and causal discovery. As a foundation for making an LLM-based causality elicitation workflow viable, we explicitly incorporate:
In Step (v), we apply causal discovery methods to the document--event incidence matrix to extract hypothesized causal links. We then aim to externalize, as a graph, the causal structure that an LLM can plausibly posit.
Thus, our contribution is to propose an LLM-based causality elicitation pipeline, where (1) document generation, (2) event canonicalization, and (3) causal discovery play important roles. Our method can be used with existing documents, not only with documents generated by our pipeline. We note that the obtained graph may not reflect real-world causality and should be interpreted as causality encoded in the LLM. We expect that our method will be used for hypothesis formulation or causality-based summarization of documents.
Our method intersects several research areas, including event extraction, entity resolution and record linkage, semantic deduplication via embedding representations and clustering, text-as-data feature construction in the social sciences, and causal relation extraction and LLM-assisted causal modeling. Existing studies treat identity through event coreference and record linkage, scale semantic similarity using embeddings, and treat causality through causal relation extraction and LLM-assisted causal modeling. We connect these lines by placing event canonicalization (i.e., representation consistency of event variables) as an explicit upstream module, constructing a stable document--event matrix, and then connecting it to causal discovery. Through this connection, from variable identity to matrix construction to causal discovery, we aim to present causal hypotheses assumed by an LLM as inspectable and comparable candidate causal graphs.
Event extraction and cross-document event coreference are core NLP problems Cybulska2014usinga. Our event canonicalization is closely related in that it clusters event mentions, but our objective is different: to obtain a stable variable set for downstream matrix construction.
Event unification is closely related to entity resolution and record linkage Fellegi1969atheory,Elmagarmid2007duplicaterecord,Christen2012datamatching. Our embedding-first canonicalization instantiates this idea for textual event names, and the incremental LLM-assisted method maintains and updates a canonical registry. Because identity granularity is analyst-dependent, we treat normalization and equivalence as a policy knob.
Regarding clustering based on semantic similarity, prior work uses sentence embeddings and scalable retrieval to support semantic matching at scale Reimers2019sentencebert,Johnson2017billionscale. We use embeddings and clustering for merging, and use the LLM for naming and adjudicating borderline cases, to construct a human-readable canonical vocabulary that can be treated as variables downstream.
The text-as-data tradition studies document-to-feature transformations and stresses validation and interpretability Grimmer2013textas,Gentzkow2019textas. Our document--event binary incidence matrix uses event phrases as features, and canonicalization reduces redundancy and improves stability.
Causal relation extraction and causal knowledge base construction are widely studied Yang2022asurvey,Hassanzadeh2020causalknowledge,Radinsky2012learningcausality. Recent work also explores LLMs for causal discovery/modeling support, and causal reasoning Liu2025largelanguage,Ma2025causalinference, and system-oriented work organizes LLM-generated causal fragments at scale Mahadevan2025largecausal. In contrast, we focus on variable fragmentation caused by surface variation in event expressions, and connect canonicalization to causal discovery through a stable document--event matrix.
Causal inference in economics and policy evaluation is built on frameworks that address confounding and counterfactuals Pearl2009causalitymodels,Imbens2015causalinference. We treat the document--event matrix constructed from LLM-generated documents as observational data and apply multiple discovery algorithms to obtain candidate graphs as hypotheses about the dependencies assumed by the LLM. Because the resulting graph does not guarantee real-world causality, any final causal claim requires external data, identification assumptions, and expert judgment Pearl2009causalitymodels.
Assume we have $N$ documents, including ones generated from LLMs. Let the list of event mentions extracted from document $i$ be: \[ E_i = [e_{i,1}, e_{i,2}, \ldots, e_{i,|E_i|}], \] where each $e_{i,k}$ is a short text string describing an event such as a policy action, market movement, or scenario assumption.
Let $\mathcal{V}_{\mathrm{raw}}$ be the set of unique raw event strings across all documents and let $M = |\mathcal{V}_{\mathrm{raw}}|$. Define a document--event binary matrix ${\bm{X}}\in\{0,1\}^{N\times M}$ by \[ X_{i,m}=\mathbb{I}[\text{raw event } v_m \text{ appears in document } i]. \] In practice, $M$ can be large and redundant due to surface-form variation.
\paragraph{Goal.} Construct a mapping from raw event strings to canonical event labels: \[ f:\mathcal{V}_{\mathrm{raw}} \rightarrow \mathcal{V}_{\mathrm{canon}}, \] so that semantically equivalent (or policy-equivalent) raw events map to the same canonical label. Using $f$, we aim to:
Our goal is to externalize causal hypotheses implicit in LLM-generated narratives as an interpretable set of variables (canonical events) and candidate causal graphs. At a high level, the pipeline is:
In this section, we focus on canonicalization and matrix construction, and then connect them to causal discovery.
Given a topic $T$, we sample $N$ analytical documents from an LLM (e.g., $N=100$). In a reference implementation, we prompt the model to behave as an analyst and to produce detailed narratives grounded in concrete events and mechanisms. Generated documents are stored per topic to enable reproducibility and comparison across runs. Below is a prompt example.
For each document $d_i$, we extract a list of event phrases using an LLM. In practice, LLM outputs may vary in format (JSON arrays, Python-like lists, newline- or comma-separated bullets). A robust extraction layer normalizes outputs into list[str] by attempting, in order, (i) JSON parsing, (ii) Python literal parsing, and (iii) fallback splitting by newlines, commas, and bullet markers. This yields event lists $\{E_i\}$. Below is a prompt example.
Pairwise LLM matching across a large vocabulary is expensive. We therefore adopt an embedding-first strategy: compute embeddings for event mentions, cluster them, and then use an LLM to name each cluster with a human-readable canonical label.
First, we flatten all event mentions and create an order-preserving unique list $\{u_1,\dots,u_M\}$ (e.g., via unique_preserve_order). Embed each $u_m$ using an embedding model (e.g., text-embedding-3-large) and compute embeddings in batches for throughput.
\emergencystretch=2em Second, given a maximum number of clusters $K_{\max}$ (e.g., 30), apply MiniBatchKMeans with $K=\min(K_{\max},M)$. To approximate cosine similarity, we L2-normalize embeddings before applying Euclidean KMeans. Below is a prompt example.
Third, for each cluster $c$, choose the top $m$ examples (e.g., 5) that are closest to the cluster centroid. With normalized embeddings, inner products correspond to cosine similarity, enabling efficient ranking.
Finally, provide representative examples to an LLM and request exactly one canonical event phrase (at temperature 0). This yields a human-readable canonical vocabulary.
Once the canonicalization map $f$ is available, the document--event matrix can be aggregated deterministically. Group raw columns by their canonical labels and merge them by a logical OR (elementwise maximum): \[ Z_{i,c} = \max_{m \in \mathcal{G}(c)} X_{i,m}, \] where $\mathcal{G}(c)$ denotes the set of raw-event columns mapped to canonical event $c$. The interpretation is that document $i$ contains canonical event $c$ if it contains any of its raw variants.
Using the aggregated document--event matrix ${\bm{Z}}$, we estimate candidate causal graphs via causal discovery. In a reference implementation, we apply:
Estimated graphs are visualized (e.g., via Graphviz, via pydot) and stored per topic.
This section provides two case studies. In the first case study, we investigate the impact of President Trump's policies on Japan's economy after 2026. In the second case study, we investigate the impact of U.S. investment in AI on gold prices.
We illustrate the pipeline on a forward-looking policy topic with interacting mechanisms:
Using the analyst-style generation prompt (January 2026 perspective), we sample $N=100$ analytical documents from an LLM under this topic. From each document, we extract a list of meaningful events, such as policy actions, decisions, shifts in outlook, market and FX states, and risk perceptions. We then canonicalize event mentions using the embedding-first procedure (Section (ref)) with a maximum cluster count $K_{\max}=30$. After LLM-based cluster naming and the removal of non-informative columns (all-0 or all-1), we retain $C=30$ canonical events and construct the document--event incidence matrix ${\bm{Z}}\in\left\{0,1\right\}^{100 \times 30}$.
Table (ref) reports the canonical event set for this run. The events span four recurring thematic blocks: (i) tariff leverage and managed trade (broad tariffs, Section 232, universal baseline tariffs, purchase-commitment deals), (ii) technology and security restrictions (export controls, technology and data-component controls, semiconductor alignment), (iii) macro--financial and FX adjustment (JPY volatility, intervention, currency-manipulation rhetoric, BoJ trade-offs), and (iv) energy and terms-of-trade shocks (sanctions and shipping costs, LNG agreements, oil-price hedging).
First and foremost, we estimate a causal structure from ${\bm{Z}}$ using the PC procedure (Figure (ref)). The displayed PC subgraph highlights three mechanisms that repeatedly appear in the narratives:
In addition to PC, we also estimate GES and LiNGAM on the same incidence matrix (not shown in the main paper). Because our inputs are binary incidence variables constructed from narrative samples, we treat edge orientation cautiously. Qualitatively, the alternative estimators tend to preserve the same module-level structure observed in PC: (i) a relocation and FDI adjustment module tied to trade pressure and localization requirements, and (ii) a technology and security restriction bundle tied to procurement nationalism and broader supply-chain and cost shocks. Differences primarily appear in how the methods orient edges among closely related policy instruments and among FX-related variables (Events 7, 25, 27, 29, 30), which are often highly correlated within narrative samples.
In summary, this 30-event run illustrates that a finer canonical vocabulary can make the externalized hypothesis graph more interpretable. Distinct U.S. leverage instruments (broad tariffs, export controls, Buy-American procurement, rules-of-origin tightening) map into distinct Japan-side adjustment channels (FDI relocation, tariff-exemption negotiation, defense and host-nation concessions), together with a monitoring sink that captures the forward-looking nature of post-2026 scenario narratives. Overall, the PC provides a conservative summary of the conditional-dependence structure in the LLM-sampled scenario distribution, suitable as a front-end hypothesis map for domain-expert review. Indeed, the existence of analogous arguments \footnote{ \url{https://www.jri.co.jp/MediaLibrary/file/report/viewpoint/pdf/15710.pdf}; \url{https://www.jetro.go.jp/biz/areareports/special/2026/0101/0e88a5272a2374f5.html}} in the economic literature lends credibility to the three mechanisms proposed by the LLM.
We provide a second case study in a macro--financial topic where multiple mechanisms plausibly coexist:
Using the same analyst-style document-generation prompt, we sample $N=100$ English analytical documents under this topic. We extract per-document event lists, canonicalize event mentions using the embedding-first procedure (Section (ref)), and construct a document--event incidence matrix. After canonical naming and the removal of non-informative columns, we obtain $C=20$ canonical events and an aggregated binary matrix ${\bm{Z}}\in\left\{0,1\right\}^{N\times C}$. Representative events include an AI capex surge, AI-driven growth narratives tied to USD strength and real yields, monitoring state variables such as TIPS and DXY, AI-chip export controls to China, Taiwan Strait tensions, and central-bank gold purchases.
First, we estimate causal structure from ${\bm{Z}}$ using the PC procedure (Figure (ref)). A key feature of the PC graph is that the canonical event describing U.S.--China tech tensions and increased gold investment acts as a hub. It receives edges from Taiwan Strait tensions, a TIPS and DXY monitoring variable that summarizes macro--financial conditions invoked in the narratives, and an AI-to-macro mechanism that links AI investment to macro conditions. In addition, the PC contains an undirected cluster among AI-chip export controls, Taiwan Strait tensions, and record central-bank purchases, indicating that these geopolitical and policy variables co-occur strongly in the scenario distribution, while their direction is not identified under the information available in ${\bm{Z}}$.
As interpretation, overall, the PC suggests two channels in the LLM-sampled narratives. The first is a macro--financial channel, in which AI investment is described as affecting growth and financial conditions, which are proxied by variables such as TIPS and DXY monitoring. The second is a geopolitical and tech-rivalry channel, in which export controls and Taiwan-related risk are discussed alongside official-sector gold accumulation. These channels converge on the gold-demand hub, providing a compact hypothesis map of how the LLM connects AI investment to gold prices.
As a summary, even with a small canonical event set ($C=20$), the PC result externalizes an interpretable structure, separating a macro--financial pathway from a geopolitical and tech-rivalry pathway, and showing where they meet in the narratives. We use this graph as a conservative summary of conditional dependence in the LLM-sampled scenario distribution, and treat it as a front-end object for domain-expert review.
First, canonicalization trades off false merges and missed merges. Whether time modifiers define identity or metadata is context-dependent and should be explicitly specified.
Second, extractions can mix events with states or mechanisms (e.g., “uncertainty increases”). A type-classification layer is a natural extension.
Third, ${\bm{Z}}$ is binary, while PC tests and LiNGAM assumptions are typically developed for continuous data. Discrete tests/SEMs and methods robust to latent confounding (e.g., FCI) are important directions.
Also, our incidence representation collapses within-document ordering, so edges primarily reflect conditional co-occurrence in the scenario distribution rather than temporal precedence. When temporal precedence is available, it can be incorporated as directional constraints in causal discovery, e.g., disallowing edges that violate known time ordering.
In addition, LLM-generated documents can reflect omissions and prompt-dependent biases. Repeated-run stability checks and external validity assessments are needed, e.g., domain-expert review of high-confidence (run-stable) edges and validation against external datasets.
Finally, we do not verify causal truth. The graphs should be viewed as hypothesis spaces for experts to inspect, refute, and refine.
We proposed a pipeline to externalize causal structure from LLM outputs. To reduce surface-form variation, we canonicalize extracted events via embedding-based clustering with LLM-based naming and aggregate a document--event incidence matrix. We then apply PC, GES, and LiNGAM to obtain candidate causal graphs as hypothesis spaces. Future work includes methods for discrete data, latent confounding, stability analysis, external validity evaluation (e.g., domain-expert review), validation with external datasets (e.g., time-series or policy indicators), and temporal extensions.