Your Language Model is Its Own Critic
Reinforcement Learning with Value Estimation from Actor’s Internal States
Yunho Choi*, Jongwon Lim*, Woojin Ahn, Minjae Oh,
Jeonghoon Shim, Yohan Jo†
Graduate School of Data Science, Seoul National University
*Equal contribution. †Corresponding author.
Accepted to NeurIPS 2026
TL;DR
Policy Optimization with Internal State Value Estimation (POISE) learns a value
baseline from the actor's internal states, without a separate LLM-scale critic.
It uses two rollouts per prompt, allowing more distinct prompts within a fixed
rollout budget. In six-domain RL training, POISE improves overall benchmark scores
on two backbones and reduces the training variance proxy on Qwen3-4B.
What is this paper about?
Stable reinforcement learning with verifiable rewards (RLVR) requires both a reliable
value baseline and diverse prompts in each batch. GRPO estimates the baseline—the
expected reward for a prompt—by averaging rewards from multiple responses to that
prompt. Larger groups improve the estimate but leave fewer distinct prompts under a
fixed rollout budget. This can increase gradient variance, especially in multi-domain
training. PPO learns a separate policy-scale critic, at substantial training and memory cost.
POISE estimates the baseline with a lightweight probe trained on the actor's hidden
states and token entropy. The probe is updated alongside the policy. For each prompt,
we sample two independent responses and predict each response's baseline from the
other's internal states. This cross-rollout construction preserves the zero-expectation
baseline term while allowing high prompt diversity. We evaluate value prediction,
policy optimization across six domains, and the contribution of this cross-rollout design.