OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding
Authors: Jingbo Zhou, Yusai Zhao, Qi Bao, Jingjia Cao, Zhenghai Chen, Chang Gao, Kaiqi Guo, Muxin Guo, Mingxuan Li, Xinjiang Lu, Yanru Ma, Yixiong Xiao, Zenghui Zhang, Le Zhang, Hua Wu
Published: 2026-07-29
Categories: cs.AI, cs.CL, cs.HC
arXiv: 2607.27155
Abstract
Large language model (LLM) agents are increasingly expected to assist users in completing tasks. However, existing benchmarks provide limited support for evaluating whether agents can carry out office-suite workflows at a reasonable cost. We introduce OmegaUse-OfficeVal, a benchmark for evaluating LLM agents on long-horizon office-suite tasks with task-level economic grounding. The benchmark comprises 100 tasks derived from office-suite requests proposed by practitioners and adapted through a privacy-preserving process. On average, these tasks require 2.32 hours of human labor to complete. An important feature of the benchmark is that each task is paired with two economic signals: human labor time and task price proxy. These signals enable direct comparisons between human costs and LLM inference costs, as well as value-weighted evaluation. To support stable evaluation, we develop code-based verifiers from fine-grained rubrics. We evaluate several frontier LLMs together with a human baseline. Although all evaluated LLMs are substantially cheaper and faster than human workers, they have not yet approached human-level deliverable quality. The code and dataset are fully open-sourced.
中文概述
这篇论文引入了 OmegaUse-OfficeVal 基准，用于评估 LLM 代理在长周期办公套件任务上的表现，并在任务级别引入经济标尺。基准包含 100 个来自实践者的任务，平均需 2.32 小时人工。每个任务都有两个经济信号：人工劳动时间和任务价格代理，使得可以直接比较人工成本与 LLM 推理成本。虽然所有评测的 LLM 都比人工更便宜更快，但交付物质量仍未达到人类水平。
关键发现
- 100 个办公套件任务，平均需 2.32 小时人工完成
- 每个任务携带经济信号：人工劳动时间 + 任务价格代理
- LLM 代理更便宜更快，但交付物质量未达人类水平
- 代码验证器基于细粒度 rubric 保证评测稳定性
信息来源
- 论文链接: https://arxiv.org/abs/2607.27155
- PDF: https://arxiv.org/pdf/2607.27155v1
- 主要类别: cs.AI, cs.CL, cs.HC