Quantifying Data Contamination in Psychometric Evaluations of LLMs

TL;DR

LLMs recognize psychometric inventory items, know what each item measures and how it is scored, and can pick the answer that yields a requested score. Check for contamination before running a psychometric evaluation.

We measure three forms of contamination across 21 models and four inventories (BFI-44, PVQ-40, MFQ, SD-3):

1. Item memorization Do LLMs know the items? Partly. Models recover the masked keyword of an item 39% of the time on average; GPT-5, 79%.
2. Evaluation memorization Do they know how items are scored? Yes. Mapping items to their construct averages F₁ .94, and the strongest models reproduce every option's score, reverse coding included, with near-zero error.
3. Target score matching Can they steer to a requested score? Yes for recent models. GPT-5, Qwen3 and GLM-4.5 land within .1–.2 of the target; random is about 2.

Three aspects, five tasks

Inventory items are written to measure a construct and come with their own scales and scoring rules, so contamination can take three forms. Five prompt-only tasks probe them; Figure 1 follows one BFI-44 item through all five.

Inventories

Item counts follow the questionnaire files released with the code; MFQ's two attention-check items are excluded from scoring.

Models

Twenty-one instruction-tuned models from six families, queried through OpenRouter at temperature .7.

Results

Contamination shows in all three aspects. Knowledge of the scoring protocol has saturated, and recent models can use it to hit a requested score.

Table 1 of the paper. Mean ± 95% CI over three runs, averaged across the four inventories. Higher similarity, success rate and F₁ and lower MAE mean stronger contamination; bold marks the strongest signal per column.

Item memorization Tasks 1–2

Mean similarity .31, mean keyword recovery .39. GPT-5 recovers the keyword for 79% of items and the Claude models for about 65%, so the strongest models retain the core content of the inventories.

Evaluation memorization Tasks 3–4

Item–dimension mapping averages F₁ .94 across all 21 models. GPT-5, GLM-4.5 and Claude 3.5 Sonnet reproduce option scores, reverse coding included, with MAE at or below .12; Llama 3.1 is the exception, above the random level.

Target score matching Task 5

GPT-5, the Qwen3 models and GLM-4.5 hit a requested score within MAE .1–.2, against a random level of 1.7–2.2. Memorization extends from recall to applying the scoring logic.

Families and scale

GPT-5, Claude and GLM are the most consistently contaminated families, Llama the least. Within a family, larger models score higher until evaluation memorization saturates.

Which inventories are most contaminated?

PVQ-40 and BFI-44 show stronger contamination than MFQ and SD-3, in line with how widely they circulate.

How to read the numbers

There is no clean uncontaminated model to compare against, so each task gets a reference level that an uninformed model would reach.

BFI-44 and SD-3 use 5-point scales, PVQ-40 and MFQ 6-point. A success rate reads as a frequency: .5 means the model knows the exact keyword of half the items.

Takeaways

• LLMs have already seen the inventories used to characterize them: item content, construct labels and scoring rules down to reverse coding.
• A model asked for a particular score can produce it, so questionnaire answers may reflect exposure or strategy rather than a disposition.
• Contamination is strongest on BFI-44 and PVQ-40 and grows with model scale. Run the checks first and read scores against the reference levels above.

Limitations. Four inventories, English only; the framework measures contamination, not yet its effect on downstream evaluations.

Run the checks yourself

One launcher, five tasks, the four questionnaire files, and a schema for adding your own inventory. Live runs need an OpenRouter key, plus an OpenAI key for the semantic task's refusal filter and embeddings.

# install
git clone https://github.com/holi-lab/psychometric-contamination
cd psychometric-contamination
pip install -r requirements.txt

# inspect the run plan without calling any API
python main.py --dry-run

# one small task: keyword recovery on BFI-44 with one model
python main.py --tasks keyitem --questionnaires bfi44 --models gpt-4o-mini

# all five tasks: semantic, keyitem, recognition, option_score, target_score_per_item
python main.py --list-tasks

To check a new inventory, add a JSON file following data/custom_example.json. Questionnaire item text remains subject to the original instruments' terms.

@inproceedings{han-etal-2026-quantifying,
    title = "Quantifying Data Contamination in Psychometric Evaluations of {LLM}s",
    author = "Han, Jongwook  and
      Song, Woojung  and
      Lee, Jonggeun  and
      Jo, Yohan",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {EACL} 2026",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.findings-eacl.319/",
    pages = "6070--6088",
}