ArcANE: Do Role-Playing Language Agents Stay in Character at the Right Time?

TL;DR

Characters change over a story, but existing role-play benchmarks treat them as fixed personas. ArcANE (Arc-Aware Narrative Evaluation) builds a Character Arc that tracks how a character's values and relationships shift across a novel, then asks whether a role-playing language agent (RPLA) answers the same scenario differently at each phase of that arc. Across six models and six ways of supplying narrative context, conditioning on the Arc is the best mode for every model, and training on ArcANE pushes a 32B model past DeepSeek-V4-Pro.

What ArcANE measures

Prior benchmarks check whether a response is in character. ArcANE checks whether it is in character at that point in the story: the same scenario is asked at every phase of a Character Arc, and the model must answer from the right phase.

Character Arcs and probes

A Character Arc segments a character's trajectory along one psychological axis (intrapersonal or relational) into phases, each with a chapter range, a state description, and anchoring events. A probe pairs one (scenario, question) with one reference response per phase, each carrying an action, a short thought, and a knowledge cutoff. Scenarios come in three types at increasing distance from the source text.

Construction pipeline

Two chapter-level LLM streams (events and psychological states) propose candidate axes, which are reconciled, screened by an LLM-critic ensemble, and, for evaluation novels, validated by three human annotators (205 of 223 axes pass). A designer LLM then drafts one probe per (target phase, category) with phase-specific references, and each probe passes two validation rounds before entering the benchmark.

Dataset

Validated novels: Harry Potter, Anna Karenina, Don Quixote, The Count of Monte Cristo, The Autobiography of Benjamin Franklin. Harry Potter is not on Project Gutenberg, so its artifacts are omitted from the public release.

Evaluation protocol

An LLM judge (DeepSeek-V4-Flash) scores each response against the phase reference on a 1–100 scale. Novels are averaged with equal weight, and Overall is the mean over the twelve probe-type × metric cells.

Context modes

All six modes share the same prompt and differ only in the context block, truncated at the query chapter.

Results

Arc context gives the best Overall score on every one of the six models.

Six models × six context modes on the validated slice (5 novels, 25 characters, 205 arcs, 1,754 probes). Switch the view by probe type, then pick the metric for the Arc lift.

Table 2 of the paper. Arc rows are shaded, the best mode per column is bold, and Arc lift is Arc minus the strongest non-Arc mode on the underlined column.

• Arc leads everywhere. +2.2 to +8.4 Overall over each model's best non-Arc mode; top mode on 29 of 30 (model, novel) cells, all 30 bootstrap intervals exclude zero.
• The lift grows with distance from the text. For DeepSeek-V4-Pro: +0.5 In-Scenario, +5.2 In-World, +7.7 Out-of-World. Retrieval already covers In-Scenario; only Arc supplies the phase elsewhere.
• The lift is largest on the trajectory metric. Per-phase metrics can be gamed phase by phase; PTF rewards only sequences that move along the arc, and its Arc gap is at least as large in every category.

Low-popularity novels

On two low-popularity titles outside the training pool (He Knew He Was Right, The Odd Women), Arc is again the top mode for all six models, by +3.8 to +12.5.

Gap and lift are Arc minus the strongest non-Arc mode, reselected per cell. Probes pass the same validators as the main slice but are not annotator-audited.

Other role-playing models

The pattern holds for HER-32B and CoSER-8B/70B: every model gains on In-World and Out-of-World, and In-Scenario stays mixed.

Inside ArcANE, SFT raises Arc Overall from 50.1 to 58.4 at 32B, while DPO is what widens the In-World and Out-of-World lift (from +7.3 to +12.5).

Analysis

What in the Arc matters?

Format alone is not enough: another character's arc in the same schema loses the gain, and the trained model needs the full phase descriptions.

MixedArc swaps in another character's arc from the same novel and fails to beat Vanilla on Qwen3-32B and ArcANE-32B-DPO. ArcHint keeps only axis names and the current phase label (about 40× shorter): within ±2.6 of the full Arc for untrained models, but only half the gain for ArcANE-32B-DPO.

Memorization and character class

The Arc lift is smallest on the most popular title and larger on supporting characters than on central ones, the opposite of what memorization would predict.

The Arc gap is smallest on Harry Potter (+3.0) and largest on Anna Karenina (+5.6) and Don Quixote (+5.1). Supporting characters gain +0.3 to +2.2 more than central characters on every model.

Is the judge trustworthy?

Annotators rate 87.1% of judge verdicts plausible, human re-scores track the judge at r = 0.96, three cross-judges agree on the ranking, and PTF drops sharply when responses are shuffled or reversed.

Why does training help?

Not a register artifact: forcing Qwen3-32B into ArcANE's first-person style lowers its score (53.8 → 50.0), while ArcANE-32B-DPO holds at 56.7. DPO's adjacent-phase contrast is what sharpens phase separation.

Training arc-aware RPLAs

SFT → DPO → RLVR lifts Qwen3-32B from 50.0 to 68.2 Overall under Arc context, above DeepSeek-V4-Pro (62.3). All five checkpoints are released.

1. SFT. Imitation targets from gpt-5.4-mini under Arc context; the teacher sees the phase reference privately, so nothing leaks into the input.
2. DPO. The anchor-phase response is chosen and an adjacent-phase response to the same scenario is rejected (14,671 pairs over 2,516 probes).
3. RLVR. GRPO on the 32B DPO model with reward mean(APF, RPF, RAE)/100 from Qwen3.6-27B; evaluation uses a separate judge on the disjoint validated slice.

RLVR improves all three probe categories, recovering DPO's In-Scenario loss while keeping its In-World and Out-of-World gains. The rewarded per-phase metrics rise under all three judges; PTF does not, since each reward covers a single phase.

Released checkpoints (Hugging Face, merged weights)

Stage	8B	32B
SFT	ArcANE-8B-SFT	ArcANE-32B-SFT
DPO	ArcANE-8B-DPO	ArcANE-32B-DPO
RLVR	not trained	ArcANE-32B-RLVR

Training data: ArcANE-Data (SFT, DPO, and RL configurations).

Takeaways

• Faithful role-play means being in character at that point in the story. ArcANE makes this phase fidelity measurable.
• Arc-grounded context beats retrieval and summaries on every model, most of all beyond the source text where retrieval has nothing to return.
• Training on ArcANE's contrastive phase pairs teaches models to track a character's trajectory, not just their traits.

@misc{song2026arcaneroleplayinglanguageagents,
      title={ArcANE: Do Role-Playing Language Agents Stay in Character at the Right Time?},
      author={Woojung Song and Nalim Kim and Sangjun Song and Chaewon Heo and Jongwon Lim and Yohan Jo},
      year={2026},
      eprint={2606.05553},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2606.05553},
}