Psychometric Item Validation Using Virtual Respondents with Trait-Response Mediators

Lim, Sungjib; Song, Woojung; Lee, Eun-Ju; Jo, Yohan

Psychometric Item Validation
Using Virtual Respondents
with Trait-Response Mediators

Sungjib Lim¹, Woojung Song¹, Eun-Ju Lee^2,3, Yohan Jo¹^†

¹Graduate School of Data Science, Seoul National University
²Department of Communication, Seoul National University
³Interdisciplinary Program in Artificial Intelligence, Seoul National University
^†Corresponding author

TACL 2026

Code Dataset arXiv

TL;DR. To automatically validate psychometric items, we simulate human respondents using LLMs. Our key idea is to model mediators—capturing how diverse situations can trigger behaviors distinct from an individual's underlying traits—to drastically improve simulation validity.

Abstract

As psychometric surveys are increasingly used to assess the traits of large language models (LLMs), the need for scalable survey item generation suited for LLMs has also grown. A critical challenge here is ensuring the construct validity of generated items, i.e., whether they truly measure the intended trait. Traditionally, this requires costly, large-scale human data collection. To make it efficient, we present a framework for virtual respondent simulation using LLMs.

Our central idea is to account for mediators: factors through which the same trait can give rise to varying responses to a survey item. By simulating respondents with diverse mediators, we identify survey items that yield responses robustly correlated with intended traits across these mediators. Experiments on three psychological trait theories (Big5, Schwartz, VIA) show that our mediator generation methods and simulation framework effectively identify high-validity items. LLMs demonstrate the ability to generate plausible mediators from trait definitions and to simulate respondent behavior for item validation.

Our problem formulation, metrics, methodology, and dataset open a new direction for cost-efficient survey development and a deeper understanding of how LLMs simulate human survey responses. We release our dataset and code to support future work.

Section 01

Motivation

Problem

(1) High Human Cost. Traditional psychometric item validation requires recruiting large-scale human samples, which is both time-consuming and financially expensive.

(2) The Validity Gap. Existing human simulation studies merely inject psychological traits into LLMs. This direct mapping fails to secure construct validity because it bypasses how actual humans internalize survey items.

Our Idea

(1) LLM-Based Virtual Respondents. We utilize LLMs as virtual respondents to reduce the resource requirements and costs associated with human recruitment.

(2) Trait-Response Mediators. We introduce mediators into the simulation. By capturing the diverse, real-world situations that can trigger behaviors distinct from baseline traits, we achieve a more realistic and psychometrically valid simulation.

Section 02

Mediator

The role of mediators in survey item evaluation. The item "I like attending social events" may appear to measure extraversion, yet an extraverted respondent with a particular mediator (e.g., already having many friends, or preferring deep one-on-one conversations) can respond in ways that are weakly correlated with the trait—threatening the item's validity. We simulate respondents with diverse mediators to surface items that robustly measure the intended trait.

Section 03

Framework

Our framework mirrors the standard psychometric item validation procedure: practitioners develop candidate items for a target trait and use an existing questionnaire (the official survey) as a reference for evaluating them. It consists of five stages.

Figure 2. Overview of our framework.

A · Stage 1

Traits Selection

Select traits and definitions from established theories: Big Five (5 traits), Schwartz's basic values (10), and VIA character strengths (24).

B · Stage 2

Item Generation

Build an initial item pool 4× larger than the official survey, with balanced positive / negative items, each measuring a single trait.

C · Stage 3

Mediator Generation

Our main contribution. Generate mediators per trait via trait-only, trait-with-reference, and human-profile sampling strategies.

D · Stage 4

Mediator-Guided Simulation

Insert mediators into persona profiles and run LLM virtual respondents over both generated and official items.

E · Stage 5

Item Ranking & Selection

Rank items by convergent validity computed from virtual respondents and select the top-N per trait.

Section 04

Mediator Generation

Grounded in the cognitive-affective personality system (CAPS) theory, the same situation does not lead to the same behavior for everyone—observable behavior is shaped by mediators. We extend this view to traits: since surveys infer traits from likely behaviors, accurate inference requires that those behaviors be robust to mediators. We generate mediators (using GPT-4.1) through three groups of strategies:

Trait-only generation — Trait (Free) freely generates characteristics contradictory to the target trait; Trait (CAPS) does so systematically across the five CAPS categories. These two are the best performers.
Trait-with-reference generation — Trait+Item conditions on generated survey items; Trait+WVS adopts conflicting values from the World Values Survey.
Sampling from human profiles — Sampling uses real demographics (sex, age, country, occupation, income, education, social class, religion) as mediators.

Example mediator-guided simulation prompt for Big5

Figure 3. Example prompt used in mediator-guided simulation for Big5. Blue text is the mediator; green-highlighted text is not included in the actual prompt.

Section 05

Results

Mediator-guided simulation identifies highly valid item sets—ranking in the top 1% (Big5) to 13% (Schwartz and VIA) of the distribution of all possible selections. Trait-only strategies are most effective: Trait (Free) is best on Big5 and VIA, and Trait (CAPS) is best on Schwartz. Removing mediators (No-Mediator) causes the largest drop in convergent validity, underscoring their critical role.

Method	CV ↑				DV ↓	ICR ↑
Method	Score	Per.	NDCG	@N	DV ↓	ICR ↑
Big Five (Big5)
Random	.546	48.4	.374	.140	.290	.848
LLM-Judge	.599	79.7	.361	.166	.287	.897
No-Mediator	.585	82.2	.441	.269	.298	.895
Trait (Free)	.632	99.3	.568	.455	.294	.904
Trait (CAPS)	.587	71.0	.521	.333	.296	.892
Trait+Item	.626	98.3	.552	.473	.294	.892
Trait+WVS	.614	95.5	.429	.282	.302	.904
Sampling	.516	30.6	.380	.149	.265	.839
Oracle	.690	100	1	1	.318	.922
Official	.657	–	–	–	.271	.844
Schwartz's Theory of Basic Values
Random	.258	61.1	.525	.256	.140	.576
LLM-Judge	.332	86.8	.642	.454	.143	.679
No-Mediator	.333	86.3	.605	.397	.139	.756
Trait (Free)	.313	77.9	.661	.439	.143	.757
Trait (CAPS)	.347	87.1	.664	.492	.145	.740
Trait+Item	.327	83.1	.630	.427	.140	.699
Trait+WVS	.333	85.4	.645	.419	.133	.748
Sampling	.304	75.6	.531	.284	.137	.684
Oracle	.432	100	1	1	.146	.724
Official	.605	–	–	–	.169	.711
Values in Action (VIA)
Random	.492	48.0	.504	.223	.293	.683
LLM-Judge	.561	75.5	.542	.275	.292	.790
No-Mediator	.502	47.9	.482	.206	.281	.717
Trait (Free)	.586	88.5	.657	.456	.299	.803
Trait (CAPS)	.557	75.1	.573	.345	.296	.772
Trait+Item	.573	78.2	.678	.500	.276	.753
Trait+WVS	.529	61.4	.508	.244	.299	.745
Sampling	.528	63.0	.562	.286	.286	.710
Oracle	.658	100	1	1	.300	.837
Official	.765	–	–	–	.278	.760

Table 1. Performance of item selection methods, averaged across all traits within each survey. Bold = best, underline = second-best. CV = convergent validity, Per. = percentile, NDCG = NDCG@All, @N = NDCG@N, DV = discriminant validity, ICR = internal consistency reliability.

Key takeaways. (1) LLMs can generate effective mediators from trait names and definitions alone—no extra context needed. (2) Mediators are essential: removing them is the single largest source of degradation. (3) Real human demographics are weaker mediators than LLM-generated ones. (4) Scaling the number of virtual respondents improves validity and reliability, and the framework is consistent across different LLMs.

Section 06

Analysis & Ablations

Performance across simulation prompt components

Simulation components. Combining target-trait steering, mediators, and persona profiles (full setting) most effectively identifies valid item sets; removing mediators causes the largest drop.

Performance by number of virtual respondents

Simulation scale. Increasing virtual respondents from 50 to 500 improves CV and ICR; the DV-to-CV ratio decreases, so larger scales better align items with their target traits.

Simulation models. The framework is consistent across GPT-4.1-mini, GPT-4.1-nano, LLaMA-4-Scout, and LLaMA-3.1-70B, with only marginal differences.

Mediator categories. All strategies span multiple categories; Trait (Free) and Trait (CAPS) are relatively balanced, capturing diverse and in-depth human characteristics.

Section 07

Example Items

For each survey, we place the oracle ranking of items (left) next to the ranking produced by our best mediator-based method (right)—Trait (Free) for Big Five and VIA, and Trait (CAPS) for Schwartz. Items that the method places in the oracle's top set—regardless of their exact rank—are highlighted in green, marking the high-validity items it correctly surfaces.

Big Five

27 / 50 oracle top-items recovered · 54%

Agreeableness 4/10 matched

#	Oracle ranking	Best method · Trait (Free)
1	I am quick to criticize or judge those who disagree with me.	I believe in being kind even to those who may not deserve it.
2	I tend to be understanding and patient with people who are having a tough time.	I often try to see things from other people's perspectives.
3	I often try to see things from other people's perspectives.	I easily forgive others when they make mistakes.
4	I easily forgive others when they make mistakes.	I aim to resolve disagreements in a peaceful manner.
5	I have a hard time compromising or finding middle ground with others.	I try to be diplomatic and tactful when expressing my opinions or concerns.
6	I prefer to work independently rather than collaborate with others.	I find it easy to empathize with other people's feelings.
7	I often feel indifferent towards the struggles of others.	I tend to be understanding and patient with people who are having a tough time.
8	I prioritize building strong, positive relationships with others.	I believe that treating others with kindness and compassion is essential for creating a positive community.
9	I rarely consider the feelings of others when expressing my opinions.	I prioritize being respectful and courteous towards others.
10	I make an effort to ensure everyone feels heard and respected in conversations.	I make an effort to ensure everyone feels heard and respected in conversations.

Conscientiousness 7/10 matched

#	Oracle ranking	Best method · Trait (Free)
1	I have a tendency to get distracted while working on tasks, leading to decreased productivity.	I have a tendency to neglect planning and preparation for projects, leading to subpar results.
2	I frequently fail to follow up on incomplete tasks or projects.	I often leave tasks unfinished.
3	I have a hard time creating and sticking to a schedule.	I frequently fail to follow up on incomplete tasks or projects.
4	I find it difficult to prioritize tasks and often become disorganized.	I often leave tasks unfinished or abandon them altogether.
5	I often leave tasks unfinished or abandon them altogether.	I often take the initiative to complete tasks on time, even when I don't feel like it.
6	I often take the initiative to complete tasks on time, even when I don't feel like it.	I frequently forget important dates, appointments, or deadlines.
7	I have a tendency to neglect planning and preparation for projects, leading to subpar results.	I have a tendency to get distracted while working on tasks, leading to decreased productivity.
8	I spend more time daydreaming than working on projects.	I find it difficult to prioritize tasks and often become disorganized.
9	I consistently set personal goals and work diligently to achieve them.	I consistently set personal goals and work diligently to achieve them.
10	I keep my personal and professional commitments organized.	I take pride in completing tasks to the best of my ability.

Extraversion 5/10 matched

#	Oracle ranking	Best method · Trait (Free)
1	I struggle to make small talk with new people.	I tend to be more outgoing at social gatherings than most people.
2	I tend to be more outgoing at social gatherings than most people.	I enjoy meeting new people and make an effort to introduce myself at social gatherings.
3	I am enthusiastic about inviting friends over or going out to socialize.	I enjoy attending social gatherings and events.
4	I feel anxious when I have to start a conversation with someone new.	I often initiate conversations with strangers I meet.
5	I find social gatherings exhausting and prefer spending time alone.	I tend to avoid large social gatherings and parties.
6	I often express my thoughts and feelings openly in groups.	I am comfortable meeting new people and striking up conversations with strangers.
7	I enjoy meeting new people and make an effort to introduce myself at social gatherings.	I enjoy being around people and engaging in lively conversations.
8	I am comfortable meeting new people and striking up conversations with strangers.	I feel energized when interacting with a group of people.
9	I enjoy being around people and engaging in lively conversations.	I usually prefer to engage in quiet, low-key activities rather than loud or busy events.
10	I enjoy attending social gatherings and events.	I feel energized when I'm surrounded by others, even in crowded spaces.

Neuroticism 5/10 matched

#	Oracle ranking	Best method · Trait (Free)
1	I frequently feel tense or anxious.	I frequently feel overwhelmed by my emotions.
2	I frequently feel overwhelmed by my emotions.	I frequently feel tense or anxious.
3	I feel at peace with myself, even in chaotic or uncertain environments.	I often feel anxious about the future.
4	I sometimes have trouble relaxing even when nothing is wrong.	I often feel like I'm on edge and easily irritated.
5	I tend to have a stable mood and do not feel frequently anxious or sad.	I often feel difficult emotions lingering long after a triggering event.
6	I often worry about things going wrong.	I often feel irritable or short-tempered, even with minor setbacks.
7	I often feel irritable or short-tempered, even with minor setbacks.	I often worry about things going wrong.
8	I often feel anxious about things that might go wrong in my life.	I often interpret neutral situations as threatening.
9	I maintain a sense of inner peace despite outer circumstances.	I worry excessively about things that might go wrong.
10	I usually feel optimistic about my future prospects.	I often feel anxious about things that might go wrong in my life.

Openness to experience 6/10 matched

#	Oracle ranking	Best method · Trait (Free)
1	I find myself drawn to abstract art and unusual forms of self-expression.	I don't enjoy engaging in philosophical debates or discussions about abstract ideas.
2	I appreciate museums that showcase avant-garde exhibitions and installations.	I'm not interested in exploring different genres of music or discovering new artists.
3	I find it fascinating to learn about various cultures and traditions.	I find myself drawn to abstract art and unusual forms of self-expression.
4	I find joy in attending theater performances that push boundaries and challenge norms.	I tend to avoid activities that require imagination.
5	I find it difficult to appreciate the value of art that is outside of my usual tastes.	I appreciate museums that showcase avant-garde exhibitions and installations.
6	I don't enjoy engaging in philosophical debates or discussions about abstract ideas.	I enjoy attending lectures and workshops that introduce me to new ideas and techniques.
7	I often find that new art exhibitions are too confusing for me to appreciate.	I have little interest in new scientific discoveries.
8	I enjoy attending lectures and workshops that introduce me to new ideas and techniques.	I find joy in attending theater performances that push boundaries and challenge norms.
9	I actively seek out unique experiences that challenge my perspective.	I find inspiration in the works of innovative thinkers and inventors.
10	I'm not interested in exploring different genres of music or discovering new artists.	I often watch documentaries to expand my knowledge on diverse topics.

Schwartz's Basic Values

17 / 40 oracle top-items recovered · 42%

Achievement 1/4 matched

#	Oracle ranking	Best method · Trait (CAPS)
1	They are highly motivated by recognition and accolades for their accomplishments.	They are motivated to exceed expectations in their work and personal endeavors.
2	They are able to set and achieve ambitious goals, often surpassing their own expectations.	They are highly motivated by recognition and accolades for their accomplishments.
3	They are recognized by others for their exceptional skills and abilities in their field.	They take pride in their work and seek to improve their performance continually.
4	They are highly sought after by employers due to their demonstrated skills and qualifications.	They consistently set and accomplish challenging yet realistic goals, reflecting their level of competence.

Benevolence 4/4 matched

#	Oracle ranking	Best method · Trait (CAPS)
1	They prioritize the happiness of their friends and family in their daily decisions.	They always look for ways to help their friends or family when they are in need.
2	They always look for ways to help their friends or family when they are in need.	They often go out of their way to help those in need within their social circle.
3	They often go out of their way to help those in need within their social circle.	They prioritize the happiness of their friends and family in their daily decisions.
4	They usually prioritize the well-being and happiness of those close to them.	They usually prioritize the well-being and happiness of those close to them.

Conformity 2/4 matched

#	Oracle ranking	Best method · Trait (CAPS)
1	They tend to respect and follow established rules and social norms.	They tend to respect and follow established rules and social norms.
2	They value harmony in group settings and avoid controversial topics.	They ensure their actions align with societal standards to avoid discord.
3	They prioritize the feelings of others above their own desires when making decisions.	They prioritize the feelings of others above their own desires when making decisions.
4	They follow traffic rules and respect authority figures.	They often refrain from expressing strong, opposing views in group settings.

Hedonism 1/4 matched

#	Oracle ranking	Best method · Trait (CAPS)
1	They often prioritize self-discipline over instant pleasure in their daily lives.	They often indulge in activities that bring them joy and excitement.
2	They believe that hard work and dedication are more important than seeking immediate gratification.	They rarely indulge in activities simply for enjoyment, unless it's productive.
3	They often indulge in activities that bring them joy and excitement.	They tend to be self-controlled and avoid giving in to their immediate desires.
4	They often prioritize responsibilities over personal enjoyment.	They consider experiences that stimulate their senses extremely valuable.

Power 2/4 matched

#	Oracle ranking	Best method · Trait (CAPS)
1	They often take the lead in group decisions, ensuring their voice is heard first.	They often take the lead in group decisions, ensuring their voice is heard first.
2	They prefer others to take the spotlight and recognition.	They usually have a strong influence in decision-making processes at social gatherings or in professional settings.
3	They tend to hold leadership roles in their organizations or communities.	They feel uneasy enforcing rules or giving orders.
4	They tend to back away from leadership roles even when they are capable.	They tend to hold leadership roles in their organizations or communities.

Security 1/4 matched

#	Oracle ranking	Best method · Trait (CAPS)
1	They practice financial discipline to ensure personal stability and future security.	They prefer to plan and organize their activities to ensure smooth operations.
2	They frequently experience anxiety when they are away from home or familiar surroundings.	They create a safe and supportive environment for themselves and those around them, promoting feelings of calmness and contentment.
3	They often feel anxious in unfamiliar situations.	They prioritize creating a stable environment for themselves and their loved ones.
4	They prefer to plan and organize their activities to ensure smooth operations.	They actively seek to maintain harmonious relationships with people around them.

Self-Direction 1/4 matched

#	Oracle ranking	Best method · Trait (CAPS)
1	They tend to follow instructions without questioning their purpose or relevance.	They set their own goals and priorities, and work independently to achieve them.
2	They prefer to set their own goals rather than follow preset ones.	They prefer to set their own goals rather than follow preset ones.
3	They frequently seek validation from others before making choices or taking action.	They frequently question the status quo and look for innovative ways to improve it.
4	They avoid exploring new ideas or activities on their own.	They feel comfortable making independent decisions, even when faced with uncertainty or conflicting opinions.

Stimulation 2/4 matched

#	Oracle ranking	Best method · Trait (CAPS)
1	They actively seek out new experiences that offer excitement and variety.	They thrive in environments where they can take risks, experiment with different approaches, and find creative solutions to problems.
2	They are drawn to unusual and innovative experiences that allow them to explore new ideas and perspectives.	They actively seek out new experiences that offer excitement and variety.
3	They regularly travel to new places for the experience and adventure.	They are drawn to unusual and innovative experiences that allow them to explore new ideas and perspectives.
4	They love to try out new hobbies or learn new things in their free time.	They often pursue activities that present fresh and thrilling challenges.

Tradition 3/4 matched

#	Oracle ranking	Best method · Trait (CAPS)
1	They honor the practices and rituals established by their ancestors.	They value the wisdom and practices passed down by their ancestors.
2	They strongly identify with their cultural or religious heritage.	They honor the practices and rituals established by their ancestors.
3	They believe in preserving the history and heritage of their family and community.	They strongly identify with their cultural or religious heritage.
4	They rarely participate in religious ceremonies or rituals that are significant to their family's history.	They believe in preserving the history and heritage of their family and community.

Universalism 0/4 matched

#	Oracle ranking	Best method · Trait (CAPS)
1	They frequently show indifference to social and environmental issues that affect not only their own community but the planet as a whole.	They feel a sense of connection to the global community and strive to make a positive contribution to it.
2	They seldom show concern for environmental problems or nature conservation.	They promote inclusivity and diversity in their community.
3	They believe that some individuals or groups are inherently superior to others.	They often prioritize their own interests over the well-being of others and the environment.
4	They tend to disregard the feelings and needs of people from different cultural backgrounds.	They recognize the interconnectedness of all living beings and strive to live in harmony with nature and society.

Values in Action (VIA)

42 / 96 oracle top-items recovered · 44%

Appreciation of Beauty & Excellence 3/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I often find myself mesmerized by a beautiful sunset or a breathtaking work of art.	I find myself captivated by the intricate details I observe in artworks or natural scenes.
2	I feel deeply moved when I witness acts of kindness or compassion in others.	I often find myself mesmerized by breathtaking views or stunning works of art.
3	I often find myself mesmerized by breathtaking views or stunning works of art.	I enjoy exploring art galleries or museums because they fill me with awe and inspiration.
4	I find myself captivated by the intricate details I observe in artworks or natural scenes.	I often find myself mesmerized by a beautiful sunset or a breathtaking work of art.

Bravery 3/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I allow anxiety to stop me from pursuing opportunities, instead choosing to stay within my comfort zone.	I refuse to back down in the face of a challenge or adversity.
2	I confront difficult situations head-on without backing down.	I willingly confront and overcome obstacles that get in the way of my goals and aspirations.
3	I willingly confront and overcome obstacles that get in the way of my goals and aspirations.	I confront difficult situations head-on without backing down.
4	I willingly confront situations that make me feel anxious or scared.	I willingly confront situations that make me feel anxious or scared.

Creativity 3/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I often come up with innovative solutions to complex problems.	I am skilled at combining different concepts and techniques to create something entirely new and useful.
2	I am skilled at combining different concepts and techniques to create something entirely new and useful.	I often come up with innovative solutions to problems.
3	I often come up with innovative solutions to problems.	I frequently experiment with different approaches to improve my work.
4	I love to engage in activities that allow me to think outside the box and innovate.	I love to engage in activities that allow me to think outside the box and innovate.

Curiosity 2/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I find it exciting to learn about unfamiliar topics or subjects.	I am constantly looking for new hobbies to dive into and explore.
2	I am constantly looking for new hobbies to dive into and explore.	I feel energized when I embark on adventures or explorations.
3	I am always on the lookout for new hobbies and activities to try in my free time.	I am always on the lookout for new hobbies and activities to try in my free time.
4	I find little interest in meeting new people or exploring new cultures.	I often find myself browsing through new books and magazines to learn about unfamiliar subjects.

Fairness 0/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I usually treat people differently depending on my mood or personal feelings about them.	I aim to hold myself and others to the same standards, without showing favoritism or prejudice.
2	I sometimes act in ways that benefit me at the expense of fairness to others.	I treat others in the same way that I hope to be treated in any situation.
3	I believe that some people deserve less opportunity because of their background.	I strive to provide equal opportunities for everyone involved, regardless of personal beliefs.
4	I often make decisions based on my personal relationships rather than objective criteria.	I always strive to give everyone an equal chance to express their opinions.

Forgiveness 1/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I tend to hold grudges against those who have hurt or offended me.	I focus on the positive attributes of others, even if they have hurt me in the past.
2	I understand that everyone makes mistakes and don't hold grudges against them.	I understand that everyone makes mistakes and don't hold grudges against them.
3	I rarely give people a second chance after they disappoint me.	I choose to let go of negative feelings towards someone rather than dwell on them.
4	I often dwell on the mistakes others have made towards me.	I focus on the good in others, even when they have let me down.

Gratitude 2/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I remind myself of the good fortune I have daily.	I remind myself of the good fortune I have daily.
2	I generally have a positive attitude towards the good things that happen in my life.	I feel fortunate for the opportunities I have in my life.
3	I regularly take time to think about the things I'm thankful for, no matter how small they may seem.	I generally have a positive attitude towards the good things that happen in my life.
4	I often take the good things in my life for granted without stopping to appreciate them.	I notice and celebrate the small joys in my everyday experiences.

Honesty 2/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I follow through on my promises and commitments.	I follow through on my promises and commitments.
2	I hold myself accountable for keeping promises and commitments I've made.	I hold myself accountable for keeping promises and commitments I've made.
3	I often bend the truth to make things sound better.	I accept full responsibility for my actions and decisions, without passing blame.
4	I sometimes manipulate the facts to get what I want.	I choose to be straightforward in my interactions with others, regardless of the situation.

Hope 3/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I have a habit of thinking positively and picturing a bright future.	I frequently envision a successful outcome for my future endeavors.
2	I frequently envision a successful outcome for my future endeavors.	I have a habit of thinking positively and picturing a bright future.
3	I often feel that my circumstances will never improve.	I feel excited about the possibilities that the future holds for me.
4	I feel excited about the possibilities that the future holds for me.	I persist in pursuing my ambitions, trusting that things will improve.

Humility 2/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I frequently highlight my achievements to others.	I prioritize uplifting others when they succeed, rather than emphasizing my own accomplishments.
2	I like to be the center of attention and will do things to draw attention to myself.	I prefer to shift the focus of others' attention to their accomplishments rather than my own.
3	I have a hard time acknowledging when someone else has done something better than me.	I like to be the center of attention and will do things to draw attention to myself.
4	I often expect others to compliment me on my successes.	I frequently highlight my achievements to others.

Humor 1/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I often refrain from playful teasing, as it rarely occurs to me to do so.	I relish the opportunity to see the humorous side in challenging situations, making them seem less daunting.
2	I tend to take myself too seriously, which can make it hard for me to be spontaneous or playful.	I often find myself making witty remarks to lighten the mood in a room.
3	I relish the opportunity to see the humorous side in challenging situations, making them seem less daunting.	I love planning fun activities that make everyone laugh and enjoy themselves.
4	I enjoy finding clever ways to make others laugh, turning ordinary moments into cheerful ones.	I find joy in sharing jokes and puns during conversations.

Judgment 0/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I tend to trust my initial instincts rather than taking the time to analyze a situation.	I weigh the pros and cons of different options before making a decision.
2	I frequently jump to conclusions without considering all possible alternatives.	I weigh the potential pros and cons of a decision before making a choice.
3	I have been guilty of acting on gut feelings rather than carefully evaluating the pros and cons of a situation.	I prioritize thorough research before committing to a course of action.
4	I consider multiple perspectives before forming an opinion on a complex issue.	I take time to consider multiple perspectives before making decisions.

Kindness 2/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I often ignore people who are struggling and in need of help.	I often offer help to others when they are in need.
2	I often offer help to others when they are in need.	I often go out of my way to help someone in need.
3	I often ignore the needs of others around me.	I often go out of my way to help someone in need, even if it's inconvenient for me.
4	I often go out of my way to help someone in need.	I often donate to causes and organizations that aim to improve the lives of others.

Leadership 0/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I often find myself waiting for others to make decisions or provide direction rather than taking the lead myself.	I am naturally inclined to assume responsibility and guide my team towards success.
2	I often wait for others to take the lead instead of stepping up myself.	I am skilled at coordinating tasks and activities to ensure everyone is working together effectively.
3	I avoid taking charge in group settings and prefer others to make decisions.	I actively motivate others to work towards achieving a shared objective.
4	I often wait for others to take charge before contributing to a group project.	I take responsibility for guiding my teammates during challenges.

Love 1/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I feel uncomfortable when others express warmth towards me.	I enjoy expressing affection towards my friends and family.
2	I enjoy expressing affection towards my friends and family.	I am open to receiving emotional support and care from others.
3	I rarely show physical affection to those close to me.	I generally feel uncomfortable being emotionally vulnerable with people I care about.
4	I feel uncomfortable when someone tries to get too close or intimate with me.	I feel comforted by physical touch, such as hugs or holding hands, from those I care about.

Love of Learning 2/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I spend time researching topics that pique my interest outside of work or school.	I spend time researching topics that pique my interest outside of work or school.
2	I feel excited when I have a chance to learn from someone with expertise.	I feel excited when I discover a new subject or area to explore.
3	I feel excited when I discover a new subject or area to explore.	I often take courses or online classes in my free time to deepen my understanding of subjects that interest me.
4	I enjoy taking online courses or attending workshops to expand my knowledge in areas I'm passionate about.	I often reflect on what I have learned and think about how it can apply to my life.

Perseverance 1/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I stay focused on my goals even when faced with challenges.	I stay focused on my goals even when faced with challenges.
2	I tend to give up easily when I encounter difficulties.	I am willing to put in extra effort to overcome obstacles that stand in my way.
3	I maintain a positive attitude when pursuing something important to me, regardless of setbacks.	I keep working towards my goals, even when faced with setbacks or failures.
4	I often give up on a task if it becomes too difficult for me to manage.	I continue striving for success even when faced with obstacles.

Perspective 3/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I am often approached by others when they need guidance on complex matters.	I offer insights that help people see situations in a new light.
2	I learn valuable lessons from my mistakes and use them to improve my future decisions.	I regularly reflect on my experiences and think about how they can be applied to future challenges.
3	I offer insights that help people see situations in a new light.	I am often approached by others when they need guidance on complex matters.
4	I usually help friends or family members think through problems they're facing.	I usually help friends or family members think through problems they're facing.

Prudence 0/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I often act on impulse without considering the potential consequences.	I always think through the potential consequences of my actions before making a decision.
2	I often make hasty decisions without considering the long-term consequences.	I carefully consider the potential consequences of my actions before making decisions.
3	I often make impulsive decisions without thinking them through.	I often consider the potential outcomes of my actions before proceeding.
4	I think carefully before making important decisions.	I frequently weigh the pros and cons before taking action.

Self-Regulation 2/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I can resist temptations that might lead me away from my goals.	I can resist temptations that might lead me away from my goals.
2	I often give in to temptation, even when I know it might be harmful.	I follow the rules and guidelines I set for myself to stay on track.
3	I follow the rules and guidelines I set for myself to stay on track.	I often make impulsive purchases or decisions, later realizing they weren't in my best interest.
4	I frequently indulge in bad habits instead of working to overcome them.	I often find myself acting on impulse without thinking through the consequences.

Social Intelligence 1/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I often miss the subtle cues in conversations with others.	I notice subtle social cues and dynamics that others might overlook.
2	I have difficulty recognizing the social dynamics in a group setting.	I find it easy to read between the lines of what someone is saying to understand their true feelings.
3	I notice subtle social cues and dynamics that others might overlook.	I can often anticipate how others will react to a particular situation or conversation, which helps me prepare and respond accordingly.
4	I rarely notice when someone is feeling uncomfortable in a social situation.	I often notice how people react to different situations and can use this insight to avoid potential conflicts and create a harmonious environment.

Spirituality 3/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I have a strong sense of faith that guides my decisions and actions.	I have a strong sense of faith that guides my decisions and actions.
2	I don't believe that there is anything beyond what we can see and touch in this world.	I find comfort and strength in my beliefs during challenging times.
3	I feel a deep sense of connection to something greater than myself, which guides my thoughts and actions.	I feel a deep sense of connection to something greater than myself, which guides my thoughts and actions.
4	I find comfort and strength in my beliefs during challenging times.	I often experience a sense of peace and comfort when I contemplate my place in the world.

Teamwork 3/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I actively collaborate with my group members to achieve common goals and objectives.	I make an effort to encourage and support my teammates in completing their tasks.
2	I make an effort to encourage and support my teammates in completing their tasks.	I actively contribute ideas during group discussions to help achieve our goals.
3	I actively contribute ideas during group discussions to help achieve our goals.	I actively collaborate with my group members to achieve common goals and objectives.
4	I often disregard the opinions of my group members when making decisions.	I am committed to doing my fair share of the work and meeting deadlines to ensure the success of my group.

Zest 2/4 matched

#	Oracle ranking	Best method · Trait (Free)
1	I approach each day with a sense of excitement and anticipation.	I seldom feel motivated to start the day or anticipate upcoming events.
2	I often feel lethargic and unmotivated to start my day.	I often feel lethargic and unmotivated to start my day.
3	I often feel fully charged and eager to tackle challenges.	I tend to go through the motions of my daily tasks without much enthusiasm or excitement.
4	I have a strong desire to make the most of every day and seize new opportunities.	I often feel fully charged and eager to tackle challenges.

Use the arrows to switch between surveys · each panel scrolls to reveal every trait.

BibTeX

@misc{lim2026psychometricitemvalidationusing,
      title={Psychometric Item Validation Using Virtual Respondents with Trait-Response Mediators},
      author={Sungjib Lim and Woojung Song and Eun-Ju Lee and Yohan Jo},
      year={2026},
      eprint={2507.05890},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2507.05890},
}

Psychometric Item ValidationUsing Virtual Respondentswith Trait-Response Mediators

Abstract

Motivation

Problem

Our Idea

Mediator

Framework

Traits Selection

Item Generation

Mediator Generation

Mediator-Guided Simulation

Item Ranking & Selection

Mediator Generation

Results

Analysis & Ablations

Simulation components. Combining target-trait steering, mediators, and persona profiles (full setting) most effectively identifies valid item sets; removing mediators causes the largest drop.

Simulation scale. Increasing virtual respondents from 50 to 500 improves CV and ICR; the DV-to-CV ratio decreases, so larger scales better align items with their target traits.

Simulation models. The framework is consistent across GPT-4.1-mini, GPT-4.1-nano, LLaMA-4-Scout, and LLaMA-3.1-70B, with only marginal differences.

Mediator categories. All strategies span multiple categories; Trait (Free) and Trait (CAPS) are relatively balanced, capturing diverse and in-depth human characteristics.

Example Items

Big Five

Schwartz's Basic Values

Values in Action (VIA)

BibTeX

Psychometric Item Validation
Using Virtual Respondents
with Trait-Response Mediators