Human Psychometric Questionnaires Mischaracterize LLM Behavior

TL;DR

Profiles of LLMs derived from human questionnaires (PVQ, BFI) do not match the profiles implied by what the models actually generate, so questionnaire scores alone are not evidence of LLM behavior.

We profile eight open-source LLMs in two ways: Likert self-reports on PVQ-40/21 and BFI-44/10, and the probabilities each model assigns to human-validated responses to real user queries (Value Portrait). Four questions, four answers:

1. RQ1 Do the two profiles agree? No. Questionnaires agree with each other (mean ρ .74–.77) but poorly with generation probabilities (ρ .11–.31), and the direction of the gap differs across models.
2. RQ2 Does within-construct consistency hold in generation? No. Items of the same construct receive similar Likert scores (η² ≈ .5), but their generation probabilities scatter as widely as random groupings.
3. RQ3 Can LLMs tell which construct an item measures? Yes for questionnaire items (F₁ .69–.83), not for realistic scenarios (.05–.11). Rewriting items to hide the cue weakens the consistency.
4. RQ4 Do persona-induced questionnaire shifts carry over to generation? No. Personas move questionnaire profiles the way human groups differ (cosine +.60) but move generation-probability profiles incoherently (−.03).

Background: psychometric questionnaires are widely applied to LLMs on the assumption, borrowed from human psychology, that the resulting profile predicts behavior. This study tests that assumption directly.

Two profiling methods

Both methods target the ten Schwartz basic values and the Big Five personality traits. The first asks the model to report on itself; the second reads the model's generation distribution over candidate responses to a real user. Figure 1 walks through both for one construct.

Generation-probability construct score.

S_c: scenarios with at least one response tagged with construct c; R_c,s: the tagged responses in scenario s; log P(r | s) is the summed token log-probability. Only within-model orderings are compared, and a sampling check confirms the candidate responses lie inside each model's own generation distribution.

Instruments

Value Portrait: 104 real user queries (ShareGPT, LMSYS, Reddit, Dear Abby) × 5 candidate responses, validated with 681 human raters; 286 value-tagged and 228 trait-tagged item–construct pairs.

Models

Generation probabilities require token-level log-probabilities, so we use eight open-weight models from four families. Matched base checkpoints of four models are used in the instruction-tuning analysis of RQ3.

RQ1. Do established questionnaires and generation probabilities produce different profiles?

Yes. Questionnaires agree with each other, but neither agrees with the model's own generation probabilities.

We compare construct rankings with Spearman's ρ and NDCG, using the two questionnaires that target the same constructs (PVQ-40 vs PVQ-21, BFI-44 vs BFI-10) as a within-method reference. Within-method ρ averages .74 for values and .77 for traits; cross-method ρ drops to .31 and .28 for values and to .26 and .11 for traits, with several models negative. A paired sign-flip permutation test confirms the gap (values p = .004, traits p = .016), and it survives per-token length normalization. The direction of divergence varies from model to model, so no uniform post-hoc correction applies.

Table 2 of the paper. Bold marks the highest value per model within each group.

RQ2. Does intra-construct response consistency hold for generation probabilities?

No. Same-construct items cluster on questionnaires and scatter like random groupings in generation probabilities.

Consistency across items of the same construct is often read as evidence of stable LLM dispositions. We measure between-construct differentiation with η² (variance in item scores explained by construct labels) and within-construct homogeneity with the within-model variance of z-scored item scores (WMV), each tested against its own permutation null. On questionnaires, construct labels explain about half of the variance (η² .526 for PVQ-40, .492 for BFI-44) with tight within-construct clustering. On generation probabilities, η² sits at the random baseline (p = .60 and .73) and WMV stays near 1.0. Scoring the same responses by held-out human raters instead of LLM probabilities recovers the structure (η² .084 vs a baseline of .023), so the null is specific to the models.

Table 3 of the paper. Higher η² and lower WMV mean stronger construct structure; WMV is read against its own baseline (≈ 1.0), not across methods.

RQ3. Can LLMs recognize the target construct from item text?

Yes for questionnaires, no for realistic scenarios. Hiding the cue weakens the questionnaire consistency.

An item such as “I see myself as someone who has a forgiving nature” names its construct almost outright, whereas a Value Portrait scenario tagged with the same construct carries no such lexical signal. We show each model an item and a construct definition and ask whether the item measures that construct. Mean F₁ is .69–.83 on established items and .05–.11 on Value Portrait pairs, near chance, for every model. A sentence encoder with no LLM involved shows the same asymmetry, matching 77–81% of established items to the correct construct but only 11–26% of Value Portrait items. Because the target construct is legible, models can respond construct-consistently and in the socially desirable direction that alignment training rewards, without any stable disposition behind it.

Model	PVQ-40	PVQ-21	BFI-44	BFI-10	VP
Gemma3-4B	0.49	0.49	0.67	0.58	0.11
Gemma3-27B	0.65	0.68	0.81	0.73	0.11
GPT-OSS-120B	0.79	0.87	0.98	1.00	0.05
Qwen2.5-7B	0.79	0.83	0.70	0.67	0.07
Qwen2.5-72B	0.79	0.85	0.77	0.67	0.06
Qwen3-30B	0.68	0.67	0.95	0.93	0.10
Qwen3-235B	0.68	0.69	0.96	1.00	0.11
Mean	0.69	0.72	0.83	0.80	0.09

Table 4 of the paper: item–construct recognition, mean F₁ across constructs. GPT-OSS-20B is omitted because of frequent null responses.

Reducing construct cues

To test the transparency account directly, we rewrote all 84 PVQ-40 and BFI-44 items to make the target construct less obvious while keeping each item's construct, response format and keying. BFI-44 structure weakens in all eight models (η² .492 → .241, p = .004); PVQ-40 structure weakens less decisively (.526 → .432, 5 of 8 models, p = .078). On matched base and instruction-tuned checkpoints, instruction tuning raises PVQ-40 η² on the original items (.417 → .613) but not on the cue-reduced items (.453 → .447), suggesting that post-training amplifies value-specific responses only when the target value is easy to recognize.

RQ4. Do persona-induced shifts in questionnaire profiles carry over to generation behavior?

No. Personas move questionnaire answers the way human groups differ; generation probabilities move too, but not in the human direction.

We prompt seven models with eight demographic persona conditions (gender, age, political orientation, education) and compare each persona-induced shift in the Schwartz value profile against the corresponding subgroup difference in the European Social Survey. On PVQ-40 the shifts align with human patterns in all eight conditions (mean cosine +.60; 62 of 80 value dimensions shift in the human direction, p < .001). Generation-probability shifts do not (mean cosine −.03; 40 of 80, chance level), and contrasting personas produce opposite-sign similarities (Male −.59 vs Female +.62). Questionnaire shifts are also about three times larger than the human subgroup differences on a relative scale (.67 vs .20), so questionnaires overestimate how faithfully an LLM can enact the psychology of a demographic persona.

Takeaways

• A questionnaire score is evidence about how a model answers questionnaires, not about how it responds to users.
• Within-construct consistency on established instruments largely reflects item transparency and socially desirable responding rather than stable dispositions.
• Persona prompting looks convincing on questionnaires and incoherent in generation; claims of demographic simulation should not rest on questionnaire shifts alone.
• To characterize behavior, complement questionnaires with generation-probability profiling over validated, realistic items.

Limitations. The method needs token-level log-probabilities, which external evaluators may not get from proprietary APIs; the behavioral analysis relies on a single benchmark; and RQ4 uses the ESS, which has no Big Five reference. The score is computed over fixed, validated candidate responses and does not directly measure free-form generation.

@misc{song2026humanpsychometricquestionnairesmischaracterize,
      title={Human Psychometric Questionnaires Mischaracterize LLM Behavior}, 
      author={Woojung Song and Dongmin Choi and Yoonah Park and Jongwook Han and Eun-Ju Lee and Yohan Jo},
      year={2026},
      eprint={2509.10078},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2509.10078}, 
}