Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
44,992 characters · 17 sections · 25 citation commands
LLM Personas as a Substitute for Field Experiments in Method Benchmarking
\twocolumn[ \icmltitle{LLM Personas as a Substitute for Field Experiments in Method Benchmarking}
\icmlsetsymbol{equal}{*}
\icmlaffiliation{yyy}{Foster School of Business, University of Washington, Seattle, USA}
\icmlcorrespondingauthor{Enoch Hyunwook Kang}{[email removed]}
\icmlkeywords{Machine Learning, ICML}
\vskip 0.3in ]
\printAffiliationsAndNotice
One of the recurring lessons from machine learning is that improvements in methods accelerate dramatically when we can iterate quickly against cheap and reliable evaluation benchmarks blum2015ladder, zaharia2018accelerating,liao2021we, d2022underspecification, bommasani2023holistic, miller2024adding, abbas2025developing. The literature advances through an iterative process in which researchers propose a variant, test it against a benchmark, inspect failures, and refine. As a result, the limiting factor in progress is often the latency and reliability of the evaluation feedback loop rather than the availability of new ideas xin2018developers.
In many societal applications, such as pricing and matching policies, ad allocation, marketplace design, and behavioral interventions, the closest analogue to a benchmark is a field experiment harrison2004field, duflo2017economist, also known as A/B testing kohavi2020trustworthy. Here, competing methods can be compared by deploying them through estimating their causal effects on agreed-upon outcome metrics. Repeating such experiments across settings allows researchers to assess not only average performance but also robustness and behavior under adaptation.
However, field experiments are costly to set up and slow to run diamond1986laboratory, fisher1992arrangement, paluck2014field, samek2019advantages. Researchers typically must first collaborate with an organization, propose an intervention, negotiate approval across multiple stakeholders, and spend months implementing instrumentation and experimental infrastructure before any data are collected, and then wait to collect data until enough statistical power is gathered. Moreover, feasible experiments are often constrained by the organization’s interests, risk tolerance, and resource limitations, which restrict what methods can be tested and how frequently experiments can be repeated bandiera2011field.
Recently, LLM-based persona simulation has shown potential to serve as a cheaper synthetic field laboratory toubia2025database, peng2025mega. In LLM-based persona simulation, a large language model is conditioned on explicit persona descriptions, such as survey responses that capture demographic attributes, preferences, roles, or beliefs. This is then used to generate individual-level evaluations or responses to proposed policies, content, or system outputs. These simulated micro-level responses can then be aggregated into a single evaluation signal. Because such simulations are inexpensive and fast to run at scale, they make it feasible to test and iterate on methods without field experimentation or A/B testing.
However, whether LLM-based persona simulation can indeed reliably substitute for field experimentation or A/B testing for methodology testing remains largely unknown. This uncertainty is exacerbated by recent studies that provide negative results in applications beyond methodology testing, particularly when we ask causal questions to support external validity claims gui2023challenge, li2025llm, peng2025mega, gui2025leveraging. Manipulating prompts intended to represent a treatment can inadvertently shift other latent aspects of the scenario, inducing confounding.
Therefore, we face the following fundamental question:
In other words, under what protocol conditions is replacing humans with personas, from the benchmark's perspective, equivalent to changing only the evaluation population (as when we change from the New York population to the Jakarta population)?
At first glance, this may sound like a question that can be empirically/experimentally validated: one might test persona benchmarking by checking whether persona scores correlate with human A/B test outcomes across a set of methods. However, such a correlation cannot validate or invalidate the "drop-in substitute” claim. The claim is not about agreement of scores between personas and humans; it is about whether swapping the evaluator preserves the benchmark interface that methods optimize against. If personas simply represent a different evaluation population, then even in the best-case scenario, the induced score distribution can (and generally should) differ from the human one, just as the same method can score differently in New York than in Jakarta, so matching outcomes is neither necessary nor expected. For these reasons, the question we study is best treated as an identification question.
In this paper, we give the exact theoretical characterization of when this benchmark-level equivalence holds, by identifying two benchmark-hygiene conditions:
We show that these two conditions are jointly necessary and sufficient for swapping human evaluation for persona evaluation to be equivalent to changing only the evaluation population, e.g., changing from the New York population to the Jakarta population. In other words, swapping humans for personas is indistinguishable by the method interface from an ordinary change of the evaluation population.
In addition to this identification result, we extend the identification discussion into a notion of usefulness: once (AO)+(MB) makes persona benchmarking identification-valid (i.e., “just panel change” on the method’s interface), the remaining question is whether the induced aggregate channel \(Q_{\mathrm{pers}}(\cdot\mid w)\) is informative enough to distinguish and optimize meaningfully different methods. We formalize this via an information-based discriminability parameter (worst-case KL separation at a pre-specified resolution) and show it yields an explicit sample-complexity rule for how many independent persona evaluations (equivalently, what effective persona dataset/panel size) are required for reliable method comparison. In this sense, beyond enforcing (AO)+(MB), “persona quality” becomes a measurable budget question: is the persona panel large enough to resolve the improvements we care about?
The rest of the paper is organized as follows. Section (ref) reviews related work. Section (ref) formalizes the benchmarking setup and states the (AO) and (MB) conditions. Section (ref) gives the main identification discussion, proving that if and only if (AO) and (MB) holds, persona vs.\ human evaluation is just panel change (JPC). Section (ref) extends the identification discussion to a usefulness discussion that relates to sample complexity. The Appendix contains deferred proofs.
We discuss papers in the literature on benchmarks in AI in computer science and field experiment design that are related to the key contributions of our paper.
Aggregate-only observation (AO), a key condition for using persona simulation rather than field experiments for method benchmarking, has also appeared in the literature on benchmark gaming blum2015ladder, hardt2017climbing, feldman2019advantages, biderman2024trenches as a baseline for preventing leaderboard gaming via adaptive overfitting. In econometrics, treatments of randomized experiments formalize the evaluation protocol as an assignment-and-aggregation procedure, clarifying what is identified from the realized (often aggregated) outcomes atheyimbens2017econometrics.
Another key condition, method-blind evaluation (MB), has been experimentally motivated: people rate the same advice differently when it is labeled as AI rather than human/crowd-sourced bogert2022human, osborne2025me; many large-scale evaluation platforms implement partial blinding (e.g., anonymous pairwise comparisons) precisely to reduce provenance effects chiang2024chatbotarena. dominguezolmedo2024questioning shows that survey-style elicitation of LLMs exhibits strong ordering and labeling effects that can dominate conclusions. The econometrics literature has also shown that outcomes can shift when participants condition on contextual/provenance cues, motivating blinding and stable scoring protocols dequidt2018demand,levitt2011hawthorne.
Related to quantifying a benchmark's usefulness, madaan2024variance studied variance-focused analyses that quantify the extent to which benchmark scores fluctuate during evaluation. heineman2025signal discusses “signal vs.\ noise” perspectives and argue that benchmarks with higher signal-to-noise ratios are more reliable for model-selection decisions. Compared to these papers, we focus on proposing a benchmark-internal information measure (minimum KL separation) that directly yields sample-size scaling for reliable method comparison on the induced aggregate channel, clarifying how large a persona dataset is practically enough.
Let $\Theta$ denote the space of tunable configuration variables; this includes all controllable degrees of freedom that specify a system or a procedure (e.g., model weights, prompts/context, hyperparameters, decoding rules, tool policies, memory policies, data curation choices, or post-processing rules). A single candidate configuration is a choice $\theta\in\Theta$. A submitted artifact is what you hand to the benchmark, i.e., the externally visible object that the benchmark evaluates. We model this via an artifact map $g:\Theta\to\mathcal W$ and write \[ w = g(\theta)\in\mathcal W. \] Possible choice of the artifact space $\mathcal{W}$ encompass: (i) a single output for a fixed input, (ii) an output distribution (stochastic method), (iii) a full interaction method mapping contexts to actions/outputs, (iv) a rollout distribution of an agent interacting with tools or environments.
\paragraph{Method as a configuration optimizer.} \;We model method benchmarking as a “submit--observe” loop. A method (algorithm) may be either non-adaptive (a single-shot submission, or a fixed distribution over submissions) or adaptive (updating submissions based on past benchmark feedback). Concretely, at each round $t=1,2,\dots,T$, the method chooses a configuration $\theta_t\in\Theta$ (equivalently an artifact $w_t := g(\theta_t)\in\mathcal{W}$), submits $w_t$ to an evaluator, and receives a feedback observation $o_t$ taking values in some observation space $\mathcal{O}$. Formally, a method (or method) $\mathcal{A}$ is modeled as a (possibly randomized) procedure that induces a decision kernel $\pi_t$ over configurations at each round $t$, such that \[ \theta_t \sim \pi_t(\cdot \mid H_{t-1}, S), \] where $H_{t-1}$ is the method's observable history before round $t$, i.e., \[ H_{t-1} := \{(\theta_\tau,o_\tau)\}_{\tau=1}^{t-1} `\] and $S$ denotes any side information available before benchmarking begins (e.g., offline datasets, pretrained weights, logs, simulators).
Note that such a definition of an method contains:
We model an evaluation as a two-stage process: many micro-level judgments are first produced, and these are then compressed into a single aggregate feedback signal that the method actually observes. Under this model, an evaluation setup (or simply an evaluator) is fully specified as \[ (P,I,\Gamma,L), \] which we will treat as a primitive object throughout.
\paragraph{Panel ($P$).} A panel is a population of evaluators, either humans or LLM personas. Formally, let $\mathcal{P}$ denote the panel space and let $P$ be a distribution over $\mathcal{P}$. Each evaluation call draws a fresh panel of $L$ independent evaluators \[ p_1,\dots,p_L \overset{\text{i.i.d.}}{\sim} P. \]
\paragraph{Micro-instrument ($I$). } Given an artifact $w\in\mathcal{W}$ and a panel member $p\in\mathcal{P}$, the micro-instrument produces an individual response. Formally, this is a conditional distribution
where the micro-response space $\mathcal{Z}$ can be a Likert score, a binary preference, or a short textual judgment. Each evaluator responds independently: \[ Z_\ell \sim I(\cdot\mid w,p_\ell), \qquad \ell=1,\dots,L. \]
\paragraph{Aggregation into observed feedback ($\Gamma$, $L$).} Finally, the benchmark aggregates the $L$ micro-responses into a single observable output. This is captured by a deterministic aggregation map
where $\mathcal{O}$ is the feedback observation space. Typical examples include the mean score, a majority vote, or a pass/fail indicator. Putting these pieces together, a single evaluation call on artifact $w$ returns the aggregate
\paragraph{What's observed by the method.} Although evaluation involves panel members and micro-responses, the method never observes them. What it sees is only the induced distribution of the aggregate feedback $o\in\mathcal{O}$. The tuple $(P,I,\Gamma,L)$ therefore defines a Markov kernel on $\mathcal{O}$:
Intuitively, $Q_{P,I}(\cdot\mid w)$ is the distribution of the single observable feedback produced by the whole pipeline “sample panel $\to$ elicit micro-responses $\to$ aggregate” when the submitted artifact is $w$. All identification arguments in the sequel are necessarily about this reduced-form object.
Up to this point, we have described an evaluation setup abstractly as a tuple $(P,I,\Gamma,L)$. Now we instantiate this abstraction in the two cases we want to compare: evaluation by humans versus evaluation by LLM personas.
\paragraph{Human benchmark.} In a human benchmark, the panel distribution $P_{\mathrm{hum}}$ samples human evaluators, and the micro-instrument $I_{\mathrm{hum}}$ is the procedure that elicits a micro-response from a human (e.g., a rating, a preference, or a short written judgment). Together with the same aggregation map $\Gamma$ and panel size $L$, this induces an observable feedback kernel \[ Q_{\mathrm{hum}}(\cdot\mid w). \]
\paragraph{Persona benchmark.} In a persona benchmark, the panel distribution $P_{\mathrm{pers}}$ samples persona profiles (e.g., demographic or attitudinal descriptors), and the micro-instrument $I_{\mathrm{pers}}$ is implemented by an LLM judge conditioned on the sampled persona. Using the same aggregation $\Gamma$ and panel size $L$ yields a second observable feedback kernel \[ Q_{\mathrm{pers}}(\cdot\mid w). \]
\paragraph{What matters for the method.} Although these two pipelines differ internally (humans vs.\ personas; human judgments vs.\ LLM judgments), the method only interacts with each benchmark through the induced distribution of the aggregate feedback. In other words, for the method, the relevant objects are precisely the two reduced-form kernels $Q_{\mathrm{hum}}(\cdot\mid w)$ and $Q_{\mathrm{pers}}(\cdot\mid w)$.
In asking the key question in this paper, “when can we treat persona evaluation as a clean benchmark interface for comparing methods?”, we need to discuss two “benchmark hygiene” conditions that clarify what information the method does (and does not) get access to, and whether the benchmark behaves like a well-defined environment independent of who is playing. They are not substantive modeling assumptions about humans or LLMs; they are assumptions about what information the benchmark reveals and how the evaluator behaves as an environment.
Intuitively, (AO) says the method sees exactly what a standard leaderboard would show: one score (or label) per submission. This prevents method's gaming behavior that relies on recognizing individual panelists/personas or exploiting micro-level structure that would be invisible in the intended benchmark interface.
Before discussing the next assumption, we define a probability measure $\mathbb{P}^{\mathcal{A}}$: for a fixed benchmark implementation, running an method $\mathcal{A}$ induces a probability measure $\mathbb{P}^{\mathcal{A}}$ over the interaction transcript $(w_1,o_1,w_2,o_2,\dots)$. Under (AO), we can define the method's interaction transcript after $t-1$ rounds as \[ \widetilde{H}_{t-1}:=\{(w_\tau,o_\tau)\}_{\tau=1}^{t-1}, \] and its information before choosing $w_t$ is the $\sigma$-field \[ \mathcal I_{t-1}:=\sigma(S,\widetilde{H}_{t-1}). \]
Intuitively, (MB) is the minimal condition for calling this evaluation setup a benchmark environment at all: the evaluator should not care about the identity of the training procedure or other metadata, and care only about what was submitted (the artifact). In other words, the benchmark interaction is fully summarized by the reduced-form kernel $Q(\cdot \mid w)$, which is fixed across methods.
From the method's point of view, each benchmark is a black box: it takes an artifact $w\in\mathcal{W}$ and returns a random aggregate feedback value $o\in\mathcal{O}$. All the internal structure (panel draws, micro-judgments, and aggregation) has already been compressed into the reduced-form kernels \[ Q_{\mathrm{pers}}(\cdot\mid w) \quad\text{and}\quad Q_{\mathrm{hum}}(\cdot\mid w), \] defined in (ref). In this section, we utilize this intuition to answer to the following question:
We first formalize what it means to “only change the evaluation population” in the panel--instrument--aggregation model (a literal panel change), and then define the corresponding interface-level notion that is relevant for adaptive benchmarking methods (just panel change, JPC). We then show that for human vs.\ persona benchmarking, JPC holds if and only if two benchmark-hygiene conditions hold: aggregate-only observation (AO) and method-blind evaluation (MB).
This yields an identification result: under (AO)+(MB), swapping humans for personas is indistinguishable on the method interface from an ordinary change of evaluation population (even though the internal micro-instrument may differ); conversely, if either condition fails, the swap can change the interface in ways that go beyond “panel change.”
Definition (ref) captures the classical “same survey, different respondents” intuition: two benchmarks has the same instrument (the question/rubric and how responses are generated), the same aggregation rule, and the same sample size, but we sample from a different population (e.g., one from New York and one from Jakarta). That is, the protocol is unchanged except for the distribution. over who evaluates.
Swapping humans for personas is not literally a panel change in this narrow sense because the micro-instrument is implemented differently (humans vs.\ an LLM judge conditioned on a persona). This motivates an interface-level notion that asks whether the swap behaves like a panel change to any method that can adapt to benchmark feedback; the following Just panel change (JPC) definition serves the role.
Definition (ref) is intentionally method-centric: it quantifies over all (possibly adaptive) methods and asks whether, from their perspective, the benchmark is an oracle channel that depends only on the current submission $w_t$, with no additional side channels or method-dependent behavior. Crucially, JPC does not require $Q=Q'$ (scores need not match across humans and personas); it only requires that the form of the interaction is preserved and that the swap can be summarized entirely by replacing one artifact-to-score kernel by another. This is precisely the sense in which swapping evaluators should resemble a “panel change” rather than a change in the rules of the game; the following Lemma (ref) and (ref) formalize the mathematical equivalence.
Lemma (ref) says that, if you literally change only the panel distribution $P$ in the panel--instrument--aggregation model, then the observable transcript laws change only by replacing the reduced-form kernel $Q_{P,I}$ by $Q_{P',I}$, i.e., JPC holds. Lemma (ref) says the converse at the interface level: any JPC swap can be realized exactly as a literal panel change in a (possibly abstract) benchmark representation. Thus Definition (ref) is mathematically equivalent to “literal panel change” as an interface-level notion (i.e., up to equality of transcript laws for every adaptive method).
The next subsection provides a protocol-level characterization of exactly when that interface property holds.
At a technical level, JPC is defined by a factorization of the transcript laws through an artifact-only kernel. This makes the main theorem appear almost tautological: if we assume conditions that guarantee exactly such a factorization, we recover JPC. The point of stating the equivalence is not to rebrand a definition, but to (i) express JPC in terms of two concrete and auditable benchmark-hygiene requirements (what the benchmark reveals, and whether it is provenance-blind), and (ii) cleanly separate the two ways persona benchmarking can fail as a substitute for field experiments: leakage of micro-level information (violating AO) and method/provenance dependence (violating MB).
Lemma (ref) also highlights two distinct failure modes:
\paragraph{If (MB) fails, the benchmark is not a well-defined oracle environment.} When (MB) fails, there is no single kernel $Q(\cdot\mid w)$ that governs the returned score across methods. Equivalently, the evaluator's behavior depends on provenance/identity or on interaction history in a way that is not summarized by the submitted artifact. In such a case, the benchmark cannot be treated as an artifact-only oracle channel, so “panel change” is not an identified description of the swap. See Appendix (ref) for when MB can be justified.
\paragraph{If (AO) fails, the method interface changes (even if aggregate scores look the same).} If the protocol reveals micro-level information (panel identities, raw votes, ordering, etc.), then two evaluator implementations can induce the same aggregate kernel on $\mathcal O$ but still be distinguishable and exploitable by an adaptive method. Hence the observable interaction is not characterized by the aggregate channel alone. See Appendix (ref) for when AO fails.
As we discussed in section (ref), (AO)+(MB) characterizes when and only when persona benchmarking is a valid benchmark, satisfying (JPC). But validity, which is an identification argument, is not necessarily equivalent to usefulness. In this section, we formalize when and when not a persona-based LLM simulation can be a perfectly valid, but much less useful, benchmark compared to field experiments. We show that 1) dataset size is what really matters for a persona-based benchmark, or any benchmark, to be considered a useful benchmark, and 2) how to find such a required dataset size empirically.
As a starting point, recall that, under (AO)+(MB), the method interacts with the persona benchmark only through the reduced-form kernel \[ Q_{\mathrm{pers}}(\cdot\mid w)\quad \text{on}\quad \mathcal O, \] which returns an aggregate feedback draw $o\sim Q_{\mathrm{pers}}(\cdot\mid w)$ for each submitted artifact $w$. Thus, we arrive at an important observation:
Throughout this section, we will utilize this idea to derive the minimum persona dataset size for benchmarking.
Denote \[ \mathcal W_0 \subseteq \mathcal W \] as the region of interest for the artifacts (e.g., a tunable region of artifacts around a baseline artifact). Assume that $\mathcal O=\mathbb R$ (a scalar score), define the benchmark landscape and benchmark noise by
where $o\sim Q_{\mathrm{pers}}(\cdot\mid w).$
A useful analogy to describe the usefulness of a benchmark is the “mountain-fog” metaphor. The (unknown) benchmark landscape $w\mapsto \mu_{\mathrm{pers}}(w)$ is the mountain: it assigns to each artifact the expected aggregate score returned by the benchmark. The benchmark noise scale $w\mapsto \sigma_{\mathrm{pers}}(w)$ is the fog. This metaphor separates two different reasons a benchmark may not be useful. A benchmark can be a flat mountain, meaning that $\mu_{\mathrm{pers}}(w)$ changes only slightly across meaningfully different artifacts; or it can be noisy (thick fog), meaning that $\sigma_{\mathrm{pers}}(w)$ is large relative to the score differences the method is trying to detect.
We formalize the “mountain-fog” idea by introducing a concept we call discriminability. let $D_{\text{KL}}(\cdot,\cdot)$ be Kullback--Leibler divergence on $\mathcal O$. Also, suppose that we can define a metric $d_{\mathcal W}$ on $\mathcal W_0$.
Let $$ \mathcal{S}_r:=\left\{\left(w, w^{\prime}\right) \in \mathcal{W}_0 \times \mathcal{W}_0: d_{\mathcal{W}}\left(w, w^{\prime}\right) \geq r\right\} . $$ where $r>0$ is a resolution parameter that formalizes “minimal meaningful change” in artifacts, which is often pre-specified ex ante. The pair $(d_{\mathcal W},r)$ should be read as: “we only require the benchmark to separate artifacts that differ by at least $r$ under $d_{\mathcal W}$.” Choosing a smaller $r$ is a stricter requirement (it asks the benchmark to resolve finer changes), and it can only make discriminability harder (the infimum ranges over a larger set, so $\kappa_Q$ can only decrease). Thus $r$ should reflect the smallest change that is substantively meaningful for method development, not the smallest change that can be expressed syntactically. In Section (ref), we discuss the guidelines of figuring out $d_{\mathcal W}$ and $r$.
Fix a probability measure $\nu$ on $\mathcal{S}_r$. For $\left(w, w^{\prime}\right) \sim \nu$, define the random variable $$ U:=D_{\mathrm{KL}}\left(Q_{\text {pers }}(\cdot \mid w)\| Q_{\text {pers }}\left(\cdot \mid w^{\prime}\right)\right). $$ One conservative notion of discriminability is the quantity we call the $q$-quantile of $U$:
Note that $\text {Quantile}_q(U)$ can be rewritten as $$ \sup _{\substack{E \subset \mathcal{S}_r \\ \nu(E) \geq 1-q}} \inf _{\left(w, w^{\prime}\right) \in E} D_{\mathrm{KL}}\left(Q(\cdot \mid w) \| Q\left(\cdot \mid w^{\prime}\right)\right) $$ This implies that, for at least $1-q$ of the $r$-separated pairs (under $\nu$), the KL separation is at least $\text {Quantile}_q(U)$. If this is near zero, it means that distinct artifacts in $\mathcal W_0$ that are essentially indistinguishable through the benchmark interface, and therefore, the benchmark provides little usable feedback for methods.
In practice, however, the KL divergence is challenging to estimate empirically. The following Lemma (ref) provides a nice simplification under the canonical assumption of homoscedastic Gaussian reduced-form kernels\footnote{The stadard quantitative Berry-Esseen results give a finite-sample error bound that decays at rate $O(1 / \sqrt{L})$ for bounded summands, making this Gaussian approximation increasingly accurate as the panel size grows berry1941accuracy.}, resolving this challenge.
The right-hand side of Equation (ref) is closely related to the quantity we often call the pairwise signal-to-noise (SNR), which is defined as
This quantity is empirically estimable: it depends only on the first two moments of this kernel $Q$, namely the mean $\mu(w)=\mathbb{E}[o\mid w]$ and variance $\sigma^2(w)=\operatorname{Var}(o \mid w)$.
We define per-sample $q$-robust discriminability of a benchmark as
$\kappa_Q(q)$ has a direct operational interpretation: it is the per-sample information available to distinguish two artifacts that differ by at least $r$ in the benchmark interface.
Appendix (ref) demonstrates an example of the sample complexity analysis for a prompt-optimization-based self-improving AI system kang2025bayesian.
The definition of discriminability in (ref) depends on two user-specified design choices: a metric $d_{\mathcal W}$ on the artifact space $\mathcal W_0$ and a resolution threshold $r>0$. These are generally method- and task-specific design parameters: different method families explore different degrees of freedom in $\mathcal W$ and therefore induce different natural notions of distance and resolution. Operationally, $d_{\mathcal W}$ and $r$ determine which pairs of artifacts the benchmark is required to reliably distinguish, and therefore they determine the relevant sample complexity via Lemma (ref).
Below are practical guidelines for selecting them in a way that is both interpretable and robust.
\paragraph{Tie $d_{\mathcal W}$ to the developer's degrees of freedom.} A good default is to define $d_{\mathcal W}$ via the natural parameterization that methods actually tune. If artifacts are produced by knobs $\theta\in\Theta$ through $w(\theta)$, and there is a natural distance $d_\Theta$ on $\Theta$, one can induce a pseudo-metric on $\mathcal W$ by \[ d_{\mathcal W}(w(\theta),w(\theta')):=d_\Theta(\theta,\theta'). \] This makes $r$ interpretable as a step size in the space the method explores. Examples include the scaled Euclidean distance on continuous hyperparameters or the edit distance on a structured prompt template.
\paragraph{Choose $r$ as a minimal meaningful iteration unit.} In most benchmarking use cases, there is a natural notion of the smallest “iteration” a developer expects to be worth distinguishing. The guiding principle is that $r$ should be large enough that changes below $r$ are not worth spending benchmark budget on, but small enough that improvements developers actually seek fall above $r$"
In short, $d_{\mathcal W}$ should encode meaningful artifact differences (preferably behavioral and invariant to cosmetic changes), while $r$ should encode the smallest change that developers intend to reliably resolve. With these choices fixed, $\kappa_Q$ becomes an operationally estimable quantity, and Lemma (ref) translates it directly into the required persona data size for stable method comparison.
As a proof-of-concept, Appendix (ref) demonstrates how our discriminability-based analysis can be used to calibrate the persona dataset size needed for reliable method comparison. Concretely, we apply the procedure to the persona-simulation ad benchmark used to evaluate TextBO, a prompt-optimization based self-improving AI system kang2025bayesian, and compute the implied number of independent persona evaluations required to distinguish one-step prompt improvements at a chosen confidence level.
We characterized when LLM-persona panels can substitute for human field experiments as a benchmark interface for method development. Our main result shows that persona vs.\ human evaluation is just panel change from the method’s perspective if and only if two benchmark-hygiene conditions hold: (i) aggregate-only observation (AO) and (ii) method-blind evaluation (MB). When either fails, the benchmark can leak exploitable information or depend on provenance, breaking the interface-level equivalence.
We also separated validity from usefulness. A valid persona benchmark may still be less informative than field experimentation if the induced aggregate channel is too flat or noisy. Our discriminability $\kappa_Q$ (worst-case KL separation at resolution $r$) yields the corresponding budget scaling for reliable comparisons, on the order of $\kappa_Q^{-1}(q)\log(1/\delta)$. Thus, beyond enforcing (AO)+(MB), the practical requirement is a sufficient persona dataset.