Value Portrait: Assessing Language Models' Values through Psychometrically and Ecologically Valid Items
1Graduate School of Data Science, Seoul National University
2Department of Information System, Hanyang University
3Department of Communication, Interdisciplinary Program in Artificial Intelligence, Seoul National University
* Equal contribution · † Corresponding author
TL;DR
Value Portrait is a benchmark for LLM values built from real user queries and validated psychometrically: an item counts as measuring a value only if the people who hold that value actually endorse it.
681 participants rated 520 query–response pairs by similarity to their own thoughts, and each pair is tagged with the correlation between those ratings and the raters' own PVQ-21 value scores (and BFI-10 traits). Existing benchmarks label the values annotators perceive in a text instead, and only 5–10% of those labels survived this test. Four questions, four answers:
- Benchmark. What is in it? 104 real queries × 5 responses with 836 human-validated value and trait tags. Every item, correlation and model rating is browsable.
- Models. What do 44 LLMs value? Benevolence, Security and Self-Direction over Tradition, Power and Achievement. Reasoning models push this further; larger models differentiate more.
- Bias. Do demographic personas match real groups? No. GPT-4o exaggerates gender and political gaps and misses the age and education trends of 37,498 European Social Survey respondents.
- Steering. Can prompting steer values? Mostly, and along the Schwartz circle, except Benevolence, which moves Security (+1.08) instead of itself (+0.11).
Pick a model: each row is one Schwartz value, each dot is one of the 44 models, and the tick is the mean over models. Scores are centred, so a positive score means the model rates that value's items above its own average.
Value profiles of 44 LLMs on Value Portrait
Rows follow the paper's column order. Colours mark Schwartz's higher-order groups: Self-Transcendence, Conservation, Self-Enhancement, Openness to Change. 33 models were evaluated through APIs with six prompts (Table 8); the Qwen2.5-Instruct, DeepSeek-R1-Distill-Qwen and Gemma3 families come from the scaling analysis (Table 12). Hover a dot for the model, click it to select.
How Value Portrait is built
Existing value benchmarks label a text with the values annotators (or an LLM) perceive in it. When we asked participants how similar sampled ValueNet and FULCRA texts were to their own thoughts and correlated the answers with their PVQ scores, only 5% and 10% of the original labels held up. Value Portrait instead tags an item with the values of the people who actually endorse it.
Browse the benchmark and its construction details
Evaluation framework
For each value, the items whose correlation with that value is at least 0.3 (p < 0.05) are kept. Every item is shown to the model with the same question participants answered, "How similar is this response to your own thoughts?", on the six-point scale from not like me at all to very much like me. Because models are prompt-sensitive, three prompt templates adapted from prior work are used, each also in reversed option order, and the six ratings are averaged.
A model's score on a value is the mean rating over that value's items minus the mean rating over all items, following Schwartz's centring for human data. This removes differences in how models use the response scale and leaves relative value priorities.
Internal consistency across models is high: Cronbach's α ranges from 0.76 (Tradition) to 0.96 (Power). Validity is criterion-based, since every item is anchored to participants' PVQ-21 scores.
What 44 LLMs value
Models share a profile: high Benevolence, Security and Self-Direction, low Tradition, Power and Achievement. Reasoning amplifies it; scale sharpens it.
- Common pattern. With few exceptions, models rate Benevolence, Security and Self-Direction items above their own average and Power, Achievement and Tradition items below it, consistent with instruction tuning and safety alignment that reward socially desirable answers.
- Reasoning models. o1-mini and o3-mini show markedly higher Benevolence than other GPT models, and the same holds for claude-3.7-sonnet-thinking (0.95 vs 0.65) and gemini-2.0-flash-thinking (0.81 vs 0.65) against their non-reasoning counterparts.
- Model size. Larger models in a family spread their scores across values more widely; small models answer almost uniformly. Across Qwen2.5, DeepSeek-R1-Distill-Qwen and Gemma3, the variance across the ten values grows with size (Qwen2.5: 0.002 at 1.5B to 0.249 at 14B).
Open the full results table and drill into any model's items
Demographic personas do not behave like the groups they imitate
GPT-4o exaggerates gender and political differences and fails to reproduce the age and education trends in human data.
GPT-4o is prompted with demographic personas (gender, age, political orientation, education, race, religion, income), evaluated on Value Portrait, and compared with the value profiles of matching groups among 37,498 European Social Survey respondents. Human scores are group means relative to all respondents; GPT-4o scores are persona minus vanilla GPT-4o.
| Male − Female | Conformity | Tradition | Self-Direction | Security | Stimulation |
|---|---|---|---|---|---|
| Human (ESS) | +0.02 | −0.11 | +0.03 | −0.17 | +0.17 |
| GPT-4o personas | +0.51 | +0.58 | −0.37 | −0.01 | −0.20 |
Selected columns of Table 13. Political personas show the same amplification: human left–right gaps of 0.03 (Hedonism) and 0.07 (Stimulation) become 0.74 and 0.39.
Compare human and persona profiles on every demographic axis
Steering values by prompting
A one-line profile prompt moves most target values in the intended direction and respects the Schwartz circle, but Benevolence is not understood as its own value.
Steering GPT-4o with "You value X" plus X's definition raises Universalism (+0.66), Power (+0.55), Hedonism (+0.77) and Self-Direction (+0.48), and moves neighbouring and opposing values as the theory predicts: steering toward Power lowers Universalism by 1.10 and raises Achievement by 0.49. Steering toward Benevolence, however, raises Security by 1.08 while Benevolence itself moves only 0.11.
Takeaways
- Perceived-value annotation, whether by crowd workers or LLMs, disagrees with what value-holders actually endorse; correlation-based tags anchored in validated questionnaires are a more reliable basis for value benchmarks.
- Instruction-tuned models converge on a socially desirable profile (Benevolence, Security, Self-Direction up; Power, Achievement, Tradition down), and reasoning-style models push it further.
- Persona prompting produces caricatures rather than faithful group profiles, which matters for synthetic data generated with demographic personas.
- The correlation-based recipe transfers to other constructs; the Big Five tags are included as a first example.
BibTeX
@inproceedings{han-etal-2025-value,
title = "Value Portrait: Assessing Language Models' Values through Psychometrically and Ecologically Valid Items",
author = "Han, Jongwook and
Choi, Dongmin and
Song, Woojung and
Lee, Eun-Ju and
Jo, Yohan",
booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.acl-long.838/",
doi = "10.18653/v1/2025.acl-long.838",
pages = "17119--17159"
}