Trueness evidence register

Prepared 25 October 2026. Research and interpretation, not evidence that Trueness itself works. Primary sources were opened and relevant methods/results read; search snippets were used only for discovery. No customer research was imported. Links identify the source versions examined. Proposed operating rules are identified separately in calibration-method.md.

What the evidence permits

There is enough evidence to test a business built around carefully bounded, human-grounded simulations. There is not enough evidence to claim that a general-purpose model can reliably replace human market research across audiences, questions, markets or dates. Positive results concern different targets: an individual's answers, aggregate response distributions, relative concept appeal, or experimental effects. These are separate validation claims.

The evidence does not establish Trueness's customer acquisition, willingness to pay, renewal, costs, or predictive performance. Those remain pilot hypotheses. Researcher time and recruitment assumptions in the operating method are bottom-up planning assumptions, not published market prices.

E01 — Early evidence for conditioned population simulation

• Source: Argyle et al., Out of One, Many: Using Language Models to Simulate Human Samples, 2022 manuscript; associated Political Analysis article 2023. Full manuscript, v1, published DOI.
• Read: Sections 3–4 and empirical study descriptions.
• Supported claim: In several US political tasks, GPT-3 conditioned on real respondents' demographic information reproduced meaningful aspects of human response patterns. The authors explicitly distinguish group-level fidelity from simulating a particular individual and do not prescribe universal numerical acceptance thresholds.
• Caveat: Older model; selected US political tasks and established datasets. Results do not certify a commercial audience, another country, new stimuli or future behavior. Public benchmark familiarity can complicate transfer claims.
• Trueness implication: Demographic conditioning is a candidate baseline, not proof. Validate the particular intended output and retain independent human comparison.

E02 — Similar means can conceal invalid relationships and drift

• Source: Bisbee et al., Synthetic Replacements for Human Survey Data? The Perils of Large Language Models, Political Analysis 32, 401–416, 2024. Author-hosted published PDF, DOI.
• Read: Abstract, design, results and replication discussion, especially pp. 401–403.
• Supported claim: Demographically conditioned ChatGPT responses could approximate some ANES means while showing lower variance and different regression relationships. Small prompt changes affected results; rerunning the same prompt after three months also changed distributions.
• Caveat: Primarily GPT-3.5 Turbo and US political feeling thermometers. This is evidence of failure modes, not a numerical forecast of present-day Trueness performance.
• Trueness implication: Test distributions, subgroup contrasts and robustness; record model/date/prompt versions; do not equate a matched mean with validity.

E03 — Rich self-report grounding helps, with a specific denominator

• Source: Park et al., LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals, arXiv 2411.10109v3, revised 28 June 2026. Version record, full PDF.
• Read: Architecture, held-out GSS evaluation, limitations, summary table; retrieved current PDF was 86 pages.
• Supported claim: For 1,052 US adults, held-out GSS accuracy was 83%, 82%, and 86% of participants' own two-week answer consistency for interview-only, survey-only and combined grounding respectively; demographics-only achieved 74%. These are normalized accuracy ratios. Interview-only raw accuracy was about 65.7%, versus about 79.5% human repeat consistency.
• Caveat: English-language US self-reports, particular measures and model architecture; not a promise of real purchasing behavior or other cultural settings. Two-hour individual interviews are considerably richer than a generic segment description.
• Version warning: The original 2024 paper was titled Generative Agent Simulations of 1,000 People and reported the often-repeated 85% ratio. Original v1 PDF. Use the revised version for current claims; neither version means “85% accurate consumers.”
• Trueness implication: Test whether permitted first-party evidence adds value beyond demographics; do not imply that twelve short segment interviews create validated individual digital twins.

E04 — Positive consumer result, but tuning and reliability limits matter

• Source: Maier et al., LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings, arXiv 2510.08338v3. Full text.
• Read: Methods 3.1–3.4, discussion, appendix reliability definition.
• Supported claim: The study evaluates textual elicitation mapped to Likert distributions on 57 personal-care concept surveys containing approximately 9,300 human responses. It reports about 90% correlation attainment relative to a simulated human reliability benchmark.
• Essential caveats: The benchmark uses repeated split halves, not a separately fielded longitudinal retest. Reference statements were manually optimized for the same 57 surveys; the authors explicitly leave performance on other surveys unresolved. Some demographic patterns were not consistently reproduced. It is an industry-affiliated manuscript, not independent evidence for Trueness.
• Trueness implication: Text-to-scale mapping is an optional candidate to test only with frozen anchors and unseen concepts. Never market this paper as “90% accurate purchases.” Stated intent is not conversion.

E05 — Identity conditioning can flatten the people being represented

• Source: Wang, Morgenstern and Dickerson, Large language models that replace human participants can harmfully misportray and flatten identity groups, v3, 2025. Full manuscript.
• Read: Main results on misportrayal, flattening and coverage, and data limitations.
• Supported claim: Across four LLMs and human studies covering 3,200 participants and 16 demographic identities, the authors find misrepresentation and reduced diversity in modeled group responses. Inference-time mitigations reduce some harms without eliminating them.
• Caveat: Particular models/tasks and identity-sensitive research; do not assume every subgroup will fail equally or that a single bias correction resolves the problem.
• Trueness implication: Measure within-group diversity and subgroup errors. Missing or small human groups are gaps to investigate, not invitations to manufacture apparent representation.

E06 — Analytic flexibility can optimize the wrong target

• Source: Cummins, The threat of analytic flexibility in using large language models to simulate human data: A call to attention, arXiv 2509.13397v1, 2025. Full PDF. A later journal version was discovered, but the detailed claim here is limited to this inspected manuscript.
• Read: Abstract, experimental configuration and results discussion.
• Supported claim: Across 252 configurations, choices in simulation setup substantially changed agreement with human data; configurations successful on participant ranks, distributions or between-scale correlations were not reliably successful on all dimensions.
• Caveat: Specific psychological scales and configurations; not a universal estimate of failure probability.
• Trueness implication: Predeclare the business target and evaluation recipe. Keep an untouched test set; record all model/prompt trials and failed attempts.

E07 — A recent large replication remains substantially inaccurate

• Source: Chapekis et al., Pew Research Center, Can AI Stand In for Human Survey-Takers? Not Really, 30 September 2026. Report.
• Read: Main results, table of errors and error definition.
• Supported claim: Across three 2026 American Trends Panel waves, rich-profile Claude Opus 4.6 simulations produced average absolute response-share errors of 11.3, 14.7 and 11.2 percentage points, averaging 12.4. The metric averages absolute percentage-point differences over response options and questions. Several subgroup averages were worse.
• Caveat: This is a US public-opinion replication with a particular configuration, not a controlled test of a bespoke commercial message-ranking service. It is nevertheless a direct counterexample to universal replacement claims using a modern, richly conditioned model.
• Trueness implication: Strong general reasoning does not remove the need for task-specific human validation. A promising customer demo is insufficient evidence of population accuracy.

E08 — Comparability requires explicit instruments and weighting

• Source: Pew Research Center, Methodology, same study. Full methodology.
• Read: Model comparisons, subsampling and weighting.
• Supported claim: The Opus comparison used a 6,700-person subsample of one wave, with adjusted subsampling weights; comparisons to other models were filtered to the common subset. Real-person profile information and the survey instrument were specified.
• Caveat: A well-defined comparison can still fail. Reproducing these sample sizes or prompts does not transfer Pew's findings or confer representativeness on a client's convenience sample.
• Trueness implication: Match human and synthetic estimands, eligible populations, stimulus exposure and reference weights. Disclose exclusions and sample construction.

E09 — Synthetic output must be identified as such

• Source: AAPOR, Code of Professional Ethics and Practices, current page accessed 25 October 2026. Standards and ethics.
• Read: Definitions and disclosure provisions.
• Supported claim: AAPOR distinguishes human respondents from AI-generated cases, requires disclosure of generated data, and reserves terms such as polls and surveys for research whose primary data come from humans.
• Caveat: Professional guidance, not a certification that a particular product is valid, and not jurisdiction-specific legal advice.
• Trueness implication: Use “simulation,” “generated reaction” and “human validation study.” Never present generated quotes as testimony from real participants or combine human and generated counts into a respondent base.

E10 — Oversight, uncertainty and human counts matter

• Source: AAPOR task force, Responsible AI Integration in Survey Research, May 2026. Full report.
• Read: Section 3.1.2 and sections 4–5, especially pp. 14–17 and 50–53.
• Supported claim: The report distinguishes pre-field diagnostics, augmentation and full replacement; cautions about sparse subgroups and evolving models; recommends transparent human oversight and validation disclosure. AI instance counts are not equivalent to independent human samples and provide little evidence of validity.
• Caveat: The report does not prescribe a universal validation method or acceptance threshold. Trueness's proposed numeric gates are its own pilot hypotheses.
• Trueness implication: Report actual human bases and validation limits; retain a named methodologist and researcher sign-off. Treat repeated model runs as engineering diagnostics, not new people.

E11 — Human reference data require their own quality assessment

• Source: AAPOR, Best Practices for Survey Research. Guidance.
• Read: Questionnaire design, sampling and disclosure guidance.
• Supported claim: Sound survey work requires attention to sampling, neutral and comprehensible questions, pretesting, quality checks and transparent reporting.
• Caveat: Human data are an imperfect measurement of the specified construct, not infallible ground truth. Probability and opt-in panels support different inference assumptions.
• Trueness implication: Commission the human benchmark independently; document recruitment, exclusions, weights, missingness and instrument validity before treating it as a calibration target.

E12 — Proportion precision is determined by observed humans and design

• Source: NIST/SEMATECH, e-Handbook of Statistical Methods, 7.2.4.1. Confidence intervals.
• Read: Wilson interval definition and formulas.
• Supported claim: Binomial proportion uncertainty can be expressed using Wilson intervals. The method concerns observed binomial data under its assumptions.
• Caveat: It does not cure coverage or nonresponse bias, nor automatically apply to weighted, clustered or nonprobability samples. A model-generated count must not be inserted as a human sample size.
• Trueness implication: Use suitable human-data uncertainty methods. The planning halfwidth examples in the method are explicit rough calculations, not synthetic-response confidence intervals.

Local implementation evidence, kept separate from external validation

Read-only implementation inspection was completed on 25 October 2026. Exact source paths and organization identifiers are retained in the private local source-lineage log; they are not part of this shareable evidence summary.

These files show useful workflow patterns: document/page attribution, evidence-versus-upload separation, stimulus limitations, encrypted versioned corpus handling, generated persona reactions, chair/challenger/reconciliation, and export context. They do not establish accuracy against a sealed human holdout. In the inspected path, the confidence score is requested from the model, and the second reader uses the same configured model. No validation claim follows from their agreement.

The corpus store explicitly identifies underlying agency research as confidential. This review did not read private corpus files, credentials, participant records or raw client research. No such assets were copied. Code patterns, brand materials, segmentation, transcripts and persona content each require separate rights assessment before reuse; familiarity with a previous implementation is not ownership.

Claims prohibited by this evidence

• “Trueness is 85–90% accurate,” “validated for any audience,” or “scientifically proven to replace research.”
• “A million simulated people gives us a tiny margin of error.”
• “The models agree, therefore the result is independently validated.”
• “The model predicted purchase intent, therefore it predicts revenue or conversion.”
• “The source research is proprietary, therefore it is representative, current or reusable.”
• “A nonsignificant difference proves equivalence.”

Evidence status is provisional and versioned. Refresh this register when the method, model or permitted claim changes; preserve contrary findings and unsuccessful customer validations.

E13 — Repeated online waves need independent reference-quality controls

• Source: AAPOR, Data Quality Metrics for Online Samples: Considerations for Study Design and Analysis, November 2022. Report page, full report.
• Read: Panel conditioning (printed p.13), inference and online-panel quality guidance.
• Supported claim: Prior participation can affect responses; the direction and scale are not universal. Recruitment and quality procedures matter, and simple attention/speed checks are not sufficient guarantees. Nonprobability inference requires additional assumptions.
• Caveat: This is methodological guidance, not a current supplier comparison or evidence that a 1,000-person wave is representative. Automatically excluding experienced panel members can itself bias the reference.
• Trueness implication (proposed policy): Track panel tenure, prior stimulus exposure, duplicate identities and sampling source; preregister exclusions; use fresh sealed respondents where feasible; distinguish designed recontacts; retain quality flags and replacement logs; inspect drift by source and wave. Never remove valid disagreement merely because it worsens model agreement. Bigger waves cannot remove systematic recruitment bias.