POISE
Contents

Your Language Model is Its Own Critic

Reinforcement Learning with Value Estimation from Actor’s Internal States

Yunho Choi*, Jongwon Lim*, Woojin Ahn, Minjae Oh, Jeonghoon Shim, Yohan Jo†

Graduate School of Data Science, Seoul National University

*Equal contribution. †Corresponding author.

Accepted to NeurIPS 2026

TL;DR

Policy Optimization with Internal State Value Estimation (POISE) learns a value baseline from the actor's internal states, without a separate LLM-scale critic. It uses two rollouts per prompt, allowing more distinct prompts within a fixed rollout budget. In six-domain RL training, POISE improves overall benchmark scores on two backbones and reduces the training variance proxy on Qwen3-4B.

What is this paper about?

Stable reinforcement learning with verifiable rewards (RLVR) requires both a reliable value baseline and diverse prompts in each batch. GRPO estimates the baseline—the expected reward for a prompt—by averaging rewards from multiple responses to that prompt. Larger groups improve the estimate but leave fewer distinct prompts under a fixed rollout budget. This can increase gradient variance, especially in multi-domain training. PPO learns a separate policy-scale critic, at substantial training and memory cost.

POISE estimates the baseline with a lightweight probe trained on the actor's hidden states and token entropy. The probe is updated alongside the policy. For each prompt, we sample two independent responses and predict each response's baseline from the other's internal states. This cross-rollout construction preserves the zero-expectation baseline term while allowing high prompt diversity. We evaluate value prediction, policy optimization across six domains, and the contribution of this cross-rollout design.

Motivation

Why prompt diversity matters

At a fixed rollout budget and baseline, sampling more distinct prompts reduces the between-prompt component of gradient variance. In our controlled experiment, the reduction is larger in multi-domain training than in math-only training.

Same rollout budget, different allocation

Let B = n × m be the batch budget: n prompts with m responses each. For a fixed policy and prompt-specific baseline,

Cov(ĝ) = Σw / B + (m / B) Σb

Σw and Σb are the within-prompt and between-prompt gradient covariance matrices. Increasing m leaves the first term unchanged and increases the second. To retain prompt diversity, we need an accurate baseline from few responses per prompt. Section 2.2.

Estimated gradient variance for one versus eight rollouts per prompt in math-only and multi-domain training
Adam-moment variance proxy for OLMo-3-7B-Think-DPO on Dolci-Think-RL-7B subsets. Both allocations use the same rollout budget and fixed baselines from eight separate, precomputed rollouts. Figure 1.

Preliminary

Can the actor supply its own value estimate?

We train a lightweight probe to predict expected verifier reward from the actor's internal states. On held-out DAPO-Math prompts with Qwen3-4B, the probe achieves Pearson r = 0.870 and MAE = 0.141, compared with r = 0.676 and MAE = 0.262 for a separately trained Qwen3-4B critic.

Probe and critic value predictions against empirical Avg@8 rewards on Qwen3-4B with DAPO-Math
Value predictions are evaluated against empirical Avg@8 rewards. Both estimators use the same reward-labeled training data. The critic reads the prompt; the probe also uses features from generated trajectories. Figure 2.

Across five datasets and four backbones, including GPT-OSS-120B, the probe has lower MAE than the critic in 16 of 20 model–dataset pairs and higher Pearson correlation in 18 of 20. These comparisons evaluate value prediction on held-out prompts.

Value prediction across domains

Qwen3-4B results across all five datasets. The full comparison also includes DeepSeek-R1-Distill-Qwen-1.5B, DeepSeek-R1-Distill-Qwen-7B, and GPT-OSS-120B. Table 10.

Held-out value prediction · Qwen3-4B
DatasetMAE ↓Pearson r ↑
ProbeCriticProbeCritic
DAPO-Math0.1410.2620.8700.676
DeepScaleR0.2310.3930.6090.384
AceCoder0.2340.4990.6120.056
ToolDial0.1880.3030.8400.440
IF-RLVR0.1950.3500.6420.150

Method

Cross-rollout value estimation

A baseline predicted from a response's own features can bias its policy gradient. POISE avoids this dependence by sampling two independent responses to the same prompt and predicting each response's baseline from the other's features. The resulting REINFORCE baseline term has zero expectation.

POISE cross-rollout algorithm diagram
Each response's features predict the other response's reward. Because the responses are sampled independently given the prompt, the probe targets expected prompt-level reward. The predictions serve as cross-rollout baselines for policy optimization. Figure 3.
1

Generate two answers

The old policy samples two independent responses for each prompt.

2

Read internal signals

At a fixed layer, mean-pool the final prompt-token and reasoning-token windows separately. Average token entropy separately over all generated, reasoning, and answer tokens.

3

Compute cross-rollout advantages

For each response, predict a baseline from the other response's features and subtract it from the reward.

4

Update the policy and probe

Compute baselines before fitting the probe on the current batch. Update the policy with fixed advantages and a PPO-style clipped objective. Fit the ridge probe on paired-rollout rewards and recent trajectories.

Results

Higher overall scores across reasoning domains

After training on the six-domain GURU corpus, POISE achieves the highest overall benchmark average among the main baselines on both backbones. It exceeds DAPO by 1.35 points on Qwen3-4B and 1.96 points on OLMo-3-7B-Instruct-DPO.

Six training domains, 17 evaluation benchmarks

We train on 92K verifiable prompts with equal compute budgets and select checkpoints on the validation split. Evaluation covers the six training domains, LiveBench, and instruction following. POISE and RLOO use two rollouts per prompt; DAPO uses eight.

MathCodeScienceLogic SimulationTabular reasoning

Qwen3-4B

Overall average across 17 benchmarks (0–100 scale).

POISE
58.10
DAPO
56.75
RLOO
56.42
Base
50.16

OLMo-3-7B-Instruct-DPO

Overall average across 17 benchmarks (0–100 scale).

POISE
49.92
DAPO
47.96
RLOO
48.32
Base
34.79

All bars use the same 0–100 scale. Source: Table 1.

Comparison at the same rollout allocation. RLOO uses the other response's reward as the baseline; POISE predicts it from that response's internal states. At the same allocation of two rollouts per prompt, POISE exceeds RLOO by 1.68 points on Qwen3 and 1.60 points on OLMo. Gains vary by benchmark: POISE exceeds both DAPO and RLOO on 11 of 17 benchmarks for each backbone.

Full results: all 17 benchmarks

Scores reproduced from Table 1. The overall score averages all 17 benchmarks; each benchmark retains its own scoring rule. AIME uses Avg@32. Bold marks the highest score within each backbone, including ties. Scroll horizontally on small screens. See the evaluation protocol.

Qwen3-4B
DomainBenchmarkBaseRLOO
m = 2
DAPO
m = 8
POISE
m = 2
CodeHumanEval87.2093.9095.7394.51
CodeLiveCodeBench22.5830.8224.7335.13
CodeMBPP80.2083.4085.0085.60
LogicARC-AGI-12.253.503.503.50
LogicZebra Puzzle12.0242.3943.7741.05
MathAIME(avg@32)18.3038.4444.9043.44
MathMATH50082.4090.8092.8092.00
SimulationCodeIO11.0013.5015.5017.50
SimulationCRUXEval-I87.0086.1288.0088.75
SimulationCRUXEval-O87.1387.1280.8887.62
ScienceGPQA Diamond42.9354.0454.5556.06
ScienceSuperGPQA36.5041.1043.4044.30
Tabular ReasoningFinQA46.0544.7844.5045.78
Tabular ReasoningHiTab65.0072.1071.5073.40
Tabular ReasoningMultiHiertt49.4050.6051.4952.08
LiveBenchLiveBench39.5945.7845.2045.35
Instruction FollowingIFEval83.1880.7879.3081.70
All benchmarksAverage50.1656.4256.7558.10
OLMo-3-7B-Instruct-DPO
DomainBenchmarkBaseRLOO
m = 2
DAPO
m = 8
POISE
m = 2
CodeHumanEval15.2487.8089.6391.46
CodeLiveCodeBench6.8132.6225.8130.82
CodeMBPP53.0067.2068.8069.60
LogicARC-AGI-11.002.251.752.25
LogicZebra Puzzle18.0624.3728.5331.90
MathAIME(avg@32)18.3333.5433.8535.83
MathMATH50078.2088.8091.4092.20
SimulationCodeIO10.5019.0016.0020.00
SimulationCRUXEval-I80.1275.7570.3875.00
SimulationCRUXEval-O49.1279.7579.6280.88
ScienceGPQA Diamond40.4044.4442.4247.98
ScienceSuperGPQA26.3030.8030.3031.40
Tabular ReasoningFinQA28.6134.9734.7938.96
Tabular ReasoningHiTab36.1056.5057.6058.00
Tabular ReasoningMultiHiertt20.8332.4428.8731.55
LiveBenchLiveBench30.1834.9536.5933.54
Instruction FollowingIFEval78.5676.3478.9377.26
All benchmarksAverage34.7948.3247.9649.92

Training stability and online value estimation

On Qwen3-4B, POISE has a lower Adam-moment variance proxy over most of training, while its probe remains predictive as the policy changes.

Qwen3-4B Adam-moment variance proxy for POISE, RLOO, and rescaled DAPO
Training variance proxy. We sum Adam-moment estimates across parameters as a proxy for gradient variance and rescale DAPO to account for its advantage normalization. The proxy does not measure exact gradient variance. Figure 4.
Value-estimator MAE and Pearson correlation over Qwen3-4B training
Online prediction accuracy. We evaluate predictions every 40 training steps against empirical values from eight rollouts per validation prompt. MAE and Pearson correlation remain stable as the policy changes. Figure 5.
Additional RL comparisons: PPO and instruction following

Comparison with token-level PPO

On the main multi-domain suite, POISE scores 58.10 versus PPO's 57.24 on Qwen3, and 49.92 versus 49.33 on OLMo. PPO uses a separate full-parameter token-level critic, GAE, and one rollout per prompt. PPO is stronger on some individual benchmarks. Table 11.

Instruction-following training

Instruction following is held out from the main GURU training mixture. In a separate experiment trained on IF-RLVR, POISE achieves the best IFEval and IFBench scores among the compared methods on Qwen3-4B and DeepSeek-R1-Distill-Qwen-1.5B. Table 8.

Analysis

Which design choices matter?

We ablate the probe's input features and the cross-rollout construction to examine their effects on value prediction and policy optimization.

Input features: reasoning states and entropy improve prediction

On Qwen3-4B, reasoning states and token entropy each predict value more accurately than prompt states alone. Combining all three gives the lowest MAE and highest Pearson correlation among the tested inputs. Response length is a weaker predictor.

Input features · Table 2
Probe inputMAE ↓Pearson r ↑
Prompt hidden states only0.2340.569
Reasoning hidden states only0.1320.821
Mean entropy only0.1520.780
Response length only0.2510.494
Full estimator0.1260.838

Held-out value-prediction scores against empirical Avg@8 rewards. Table 2.

Cross-rollout baselines in policy optimization

In controlled Qwen3-4B math training, full POISE achieves higher scores than the prompt-only and same-rollout variants. Under the fixed rollout budget, these one-rollout variants see twice as many distinct prompts as POISE, so the comparison also varies prompt diversity.

Math RL · Qwen3-4B · Avg@32 over seven benchmarks
MethodRollouts / promptAvg@32 (%) ↑
Base model—25.8
Prompt-only estimator147.5
Same-rollout estimator140.7
PPO with token-level critic149.3
Full POISE250.1

All RL variants use the same total rollout budget per update. Table 3.

Probe architecture and feature extraction

The linear ridge probe matches or exceeds the tested MLPs in Pearson correlation, while some MLPs achieve lower MAE. Value prediction also works across the tested extraction layers and pooling windows. Extraction ablations; probe architectures.

Scope. We evaluate POISE under fixed compute budgets; substantially longer training runs remain untested. The estimator predicts sequence-level value. Extending it to token-level credit assignment, non-verifiable rewards, and long-horizon agentic tasks remains future work. Limitations.

BibTeX

Citation

@inproceedings{choi2026poise,
  title={Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States},
  author={Choi, Yunho and Lim, Jongwon and Ahn, Woojin and Oh, Minjae and Shim, Jeonghoon and Jo, Yohan},
  booktitle={Advances in Neural Information Processing Systems},
  year={2026},
  url={https://arxiv.org/abs/2605.07579}
}