Large Reward Models

Abstract

Reinforcement Learning (RL) has shown strong potential for improving robotic manipulation policies, yet its practical use remains bottlenecked by the difficulty of specifying reward functions that are both semantically meaningful and reusable across tasks. We propose Large Reward Models (LRMs), a framework that adapts foundation VLMs into frame-level reward generators for robot policy refinement. We specialize a state-of-the-art VLM on a multi-source dataset spanning real-world robot trajectories, human-object interactions, and simulated manipulation environments. Unlike prior approaches that mainly evaluate trajectories post-hoc, LRMs expose multiple reward interfaces from visual observations: progress estimation, task completion, and temporal contrastive comparison. Starting from an imitation-learned policy, we use these VLM-derived rewards to guide PPO refinement on held-out long-horizon manipulation tasks. Our experiments show that LRM progress rewards provide the strongest non-privileged online refinement signal, improving the IL baseline and narrowing the gap to privileged environment rewards. We further deploy progress rewards for progress-weighted behavioral cloning on four real-world manipulation tasks spanning two robot platforms, improving over SFT on all four tasks. These results suggest that modality-specific specialization of foundation VLMs can provide practical visual reward signals for both simulated policy refinement and physical robot self-improvement without hand-coded task rewards.

Method Overview

Stage 1: Multi-source Trajectory Processing. Training episodes are sampled from a balanced collection of 24 data sources: 20 Open X-Embodiment real-robot sub-datasets, 2 egocentric human-object interaction datasets (HOI4D, EgoDex), and 2 simulated benchmarks (LIBERO, RoboCasa). Each video is downsampled to 11 normalized-progress keyframes, turning the temporal order of successful trajectories into weak supervision without any manual reward labels.

Stage 2: Training Large Reward Models (LRMs) for reward generation. Starting from the same Qwen3-VL-8B-Instruct initialization, we specialize three independent reward modalities: the Temporal Contrastive Reward (\(r_{\text{cont}}\)) for relative ranking, the Absolute Progress Reward (\(r_{\text{prog}}\)) for continuous estimation, and the Task Completion Reward (\(r_{\text{comp}}\)) for terminal state anchoring. Completion and contrastive each use a separate LoRA adapter on a frozen backbone, while progress additionally updates the vision encoder and multimodal merger. No new reward tokens or architectural reward heads are introduced; a deterministic parser converts each model's structured text output into a scalar or binary reward.

Stage 3: Policy refinement with LRM-generated rewards. For simulated long-horizon manipulation (ManiSkill3), a frozen LRM is queried every \(K\) environment steps, mapping visual observations \(I_t\) and task description \(d\) into a dense reward stream that drives PPO refinement of the policy \(\pi_\phi\), initialized from an imitation-learned checkpoint. For real-world adaptation, we instead use the Absolute Progress Reward offline to weight collected rollouts for progress-weighted behavioral cloning, avoiding the need for online VLM queries during physical robot control.

Quantitative Results

Real-World Progress-Weighted Behavioral Cloning

Across four real-world tasks on two robot platforms, scoring collected rollouts with LRM-r_prog for progress-weighted behavioral cloning consistently improves over the SFT baseline, with gains ranging from 11.7 to 80.0 percentage points. Zero-shot Qwen3-VL scoring, in contrast, is task-dependent: it helps on the giraffe-in-bowl and tissue-out-of-box tasks but hurts on the can-on-cube and open-drawer tasks, indicating that LRM specialization gives a more reliable direction of improvement.