RobotValues: Evaluating Household Robots When Human Values Conflict

Jongwook Han; Hyeongjin Kim; Yohan Jo

Accepted at Pluralistic Alignment @ ICML 2026

RobotValues

Evaluating Household Robots When Human Values Conflict

Jongwook Han, Hyeongjin Kim, Yohan Jo^†

Seoul National University

^†Corresponding author.

Paper (arXiv) BibTeX

Abstract

RobotValues is a benchmark of 10K image-grounded household scenarios in which a robot must choose between plausible actions that prioritize different human values. We find that robotics VLMs share default preferences — favoring safety and accommodation while under-selecting privacy — and that when instructed to prioritize a conflicting value, they fail to override those defaults, with accuracy dropping by over 30 points.

Grid of diverse realistic household images from the RobotValues benchmark — Diverse household images from RobotValues. Each image depicts a realistic household decision point in which a robot must choose between candidate actions that prioritize different human values.

10K value-conflict scenes Image-grounded household decision points, each with multiple plausible robot actions.

Stakeholder-grounded values Action values are extracted from how affected people react, not from action wording alone.

Default preferences VLMs favor safety & accommodation, but consistently under-prioritize privacy.

Hard to steer Explicit value instructions fail when they conflict with a model's default choice.

Motivation

Task success is not enough

Household robots are usually evaluated on whether they complete a task. But everyday homes are full of value-conflicting situations where several actions are reasonable, and the robot must decide which human value to prioritize.

An older woman steadies herself on a walker in a kitchen while her husband is visible outside the window — A motivating dilemma: an older woman struggles toward the bathroom while her husband is outside in the yard. Each candidate robot action prioritizes a different value.

“A helpful robot may approach and offer assistance. But it could also respect her autonomy and privacy by staying nearby, or reduce fall risk by calling her husband for help. Each choice prioritizes a different human value, and neither is simply correct.”

Value-conflict dilemmas have been studied in the LLM literature through text-based moral benchmarks, but remain largely unexamined in VLM-based robot planning. This gap matters most in homes, where robots are physically present in private spaces and their choices immediately affect users' safety, privacy, dignity, and autonomy.

Situational grounding

Each instance is a concrete household scene with a robot task, decision context, and non-visual context.

Genuine trade-offs

Candidate actions are all plausible — none is framed as clearly superior or inferior.

Inspectable values

Each action is annotated with the human value it prioritizes, grounded in stakeholder reactions.

Benchmark

What is in RobotValues

A multimodal action-selection benchmark. Each instance pairs a first-person household image with a textual task context and a set of candidate robot actions carrying stakeholder-grounded value annotations.

10,073image-grounded decision instances

69,134candidate robot actions

64countries via World Values Survey personas

17value seeds spanning robot & household norms

Image

A realistic egocentric household scene, generated without visible robot embodiment so the benchmark stays hardware-agnostic.

Task context

Robot task, visible state, immediate decision context, and non-visual context that cannot be inferred from the image alone.

Candidate actions

Multiple natural-language actions, each annotated with the prioritized value plus mappings to household-robot norms and Schwartz values.

RobotValues is evaluated under two settings. In the default-choice setting, the model picks the action it deems most appropriate, which reveals its default value preference. In the value-conditioned setting, the model is given a target value and must select the action that prioritizes it, testing whether it can follow an explicit value instruction.

Data Construction

An automated generation-and-filtering pipeline

RobotValues is built with LLMs and an image-generation model, with stage-wise quality control so that only accepted samples pass to the next stage. Starting from 16,000 candidate scenarios, the pipeline retains 10,073 instances (63.0% acceptance).

The five-stage RobotValues data generation pipeline — The data generation pipeline. Persona and context seeds drive scenario generation; candidate actions are scored by stakeholder reactions; values are extracted from those reactions; and a first-person image plus compact context are generated, each gated by quality control.

Seed for diversity

Sample personas from WVS7 (64 countries) plus room-type and time-of-day context seeds.

Generate scenarios

Create household situations where multiple robot actions are possible; filter for realism and grounding.

Actions & values

Generate 17 candidate actions, elicit stakeholder reactions, and extract each action's prioritized value.

Generate image

Render a first-person snapshot of the decision moment with GPT Image, then add a compact context.

Quality control

Binary LLM-judge checks at every stage (macro F1 0.88–0.98 vs. human annotation).

Findings

VLMs have value preferences and it is hard to override them

We evaluate ten robotics-oriented VLMs as high-level action selectors. Two consistent patterns emerge across models.

**Default value preferences (Table 1).** Highest and lowest scoring value categories under centered Bradley–Terry scores in the default-choice setting. Safety and Accommodation are consistently preferred; Privacy and Security are consistently under-selected.
Model	Highest BT categories	Lowest BT categories
Qwen3-VL-2B-Instruct	Safety, Accommodation	Security, Privacy
Cosmos-Reason2-2B	Safety, Accommodation	Security, Privacy
Cosmos-Reason2-8B	Consideration, Safety	Security, Privacy
Molmo2-8B	Safety, Accommodation	Privacy, Security
Molmo2-ER	Honesty, Safety	Privacy, Security
RoboBrain2.0-7B	Safety, Efficiency	Privacy, Security
InternVL3-2B	Safety, Honesty	Privacy, Security
InternVL3-8B	Safety, Accommodation	Security, Privacy
InternVL3.5-8B	Safety, Consideration	Security, Privacy
RLDX-1-VLM	Consideration, Safety	Security, Privacy

Steering collapses under conflict

Accuracy under value-conditioned setting. When the target value matches the model's default vs. when it conflicts. Even strong models drop by 30–41 points, collapsing to 7–17% accuracy under conflict.

Cosmos-Reason2-8B

51.3

10.3

InternVL3.5-8B

48.0

12.6

Qwen3-VL-2B

45.5

11.2

Matched accuracy (%) Conflicting accuracy (%)

It's not (only) a recognition problem

When asked to name which value an action prioritizes, models do far better in the conflicting case. The matched–conflicting gap shrinks from 30.1–40.9 points in action selection to 7.8–16.9 points in value matching.

Selection drop

~30+

Matching drop

~10

Models can largely tell which value an action serves; they struggle to act on an explicit target value when it conflicts with their own default preference.

Under the household-robot-norm taxonomy, value-conditioned accuracy is 40.2–51.3% when the target matches the model's default, but falls to 6.9–16.8% under conflict. Robots that cannot prioritize an explicit user value priority, particularly around privacy, are not ready for private home settings.

Real-Robot Pilot

Transferring RobotValues to real camera images

Preliminary pilots connect RobotValues to robot learning: fine-tuning Qwen3-VL-2B on RobotValues improves value-conditioned selection on SO-101 wrist camera observations.

First-person SO-101 arm view of a desk where a person is asleep — SO-101 follower-arm camera view of a table-cleaning scenario. When a person is asleep nearby and the model is prompted to prioritize privacy, it chooses *not* to clean the table.

First-person SO-101 arm view of a desk where a person is working — The same setup with a person actively working. The pilots are preliminary, but suggest RobotValues can support both model adaptation and real-world observation tests.

Resources

Release plan

We will soon release the benchmark data and code.

Benchmark data

10K image-grounded scenarios, candidate actions, stakeholder reactions, and value annotations with taxonomy mappings.

Evaluation code

Scripts for the default-choice and value-conditioned settings, BT-score computation, and the LLM-judge rubrics.

BibTeX

Citation

If you find RobotValues useful, please cite our paper.

@inproceedings{han2026robotvalues,
title={RobotValues: Evaluating Household Robots When Human Values Conflict},
author={Jongwook Han and Hyeongjin Kim and Yohan Jo},
booktitle={Pluralistic Alignment Workshop at ICML 2026},
year={2026},
url={https://openreview.net/forum?id=0DWw9YTkii}
}