CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agent

Source: https://arxiv.org/abs/2510.18596

Authors: Haojia Lin, Xiaoyu Tan, Yulei Qin, Zihan Xu, Yuchen Shi, Zongyi Li, Gang Li, Shaofei Cai, Siqi Cai, Chaoyou Fu, Ke Li, Xing Sun

Published: 2025-10-21

Updated: 2025-10-21

Categories: cs.SE, cs.CV

Comment: 24 pages, 6 figures

PDF: https://arxiv.org/pdf/2510.18596v1

Abstract

Computer-using agents (CUAs) enable task completion through natural interaction with operating systems and software interfaces. While script-based verifiers are widely adopted for evaluation, they suffer from limited scalability and inability to provide step-wise assessment. Reward models offer promising alternatives, but their effectiveness on CUA evaluation remains largely underexplored. To address this gap, we present CUARewardBench, comprising four key contributions: (1) First-ever Comprehensive CUA Reward Benchmark: We introduce the first benchmark for evaluating both outcome reward models (ORM) and process reward models (PRM) on CUA tasks, enabling systematic assessment across trajectory-level and step-level evaluation. (2) Diverse, Practical and Reliable Dataset: CUARewardBench encompasses trajectories from 10 software categories and 7 agent architectures with varying performance levels (25.9%-50.8% success rates). All trajectories are expertly annotated through carefully designed protocols, with rigorous quality control to ensure reliability and practical applicability. (3) Comprehensive Analysis and Insights: Through extensive experiments across 7 vision-language models and 3 prompt templates, we reveal critical limitations of current CUA RMs, including insufficient visual reasoning capabilities, knowledge deficiencies, and the superiority of general VLMs over specialized CUA models for reward evaluation. (4) Unanimous Prompt Ensemble (UPE): Based on the insights from our comprehensive analysis, we propose UPE, a novel ensemble method that significantly enhances reward model reliability through strict unanimous voting and strategic prompt-template configurations. UPE achieves 89.8% precision and 93.3% NPV for ORM, and 81.7% precision and 85.1% NPV for PRM, substantially outperforming single VLMs and traditional ensemble approaches.

Key Findings

CUARewardBench targets reward-model evaluation for computer-using agents, covering both outcome reward models and process reward models.
The benchmark spans 10 software categories, 7 agent architectures, and trajectories with 25.9%-50.8% success rates, all expert-annotated under quality-control protocols.
Experiments across 7 vision-language models and 3 prompt templates show current CUA reward models still struggle with visual reasoning and knowledge gaps; the proposed unanimous prompt ensemble improves reliability.

中文概要

CUARewardBench 面向 computer-using agent 的 reward model 评估，同时覆盖 outcome reward model 和 process reward model。它的贡献不是单一排行榜，而是把 10 类软件、7 种智能体架构、25.9%-50.8% 不同成功率的轨迹放到一个可系统分析的框架里。论文还指出当前 CUA reward models 在视觉推理和知识上仍有短板，并用 unanimous prompt ensemble 提升 ORM/PRM 判断的可靠性。

Why it matters for AAIF

This entry is useful for tracking how agent evaluation is moving beyond single-shot task success: one paper focuses on dialogue-driven coding work, and the other focuses on reward-model assessment for GUI/software-operating agents.

Grounding

Content generated from OpenCLI arXiv metadata/abstract fetched for the canonical arXiv ID.
arXiv URL: https://arxiv.org/abs/2510.18596
PDF: https://arxiv.org/pdf/2510.18596v1