Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions?

ID: 53ed28e5
原文链接: https://arxiv.org/abs/2607.26041
PDF: https://arxiv.org/pdf/2607.26041v1
作者: Abhishek Pillai, Samir Kumar Nayak, Yuan Chen
日期: 2026-07-28
更新: 2026-07-28
分类: agents
来源类型: paper
标签: computer-use, benchmark, gui, desktop-agent, state-verification
质量评分: 4/5
抓取时间: 2026-07-30T04:18:47Z

中文导读

Desktop-Delta Bench 是面向 computer-use agent 的步级基准，测试模型能否重建动作产生的因果转换（对拒绝过时观测验证进度错误恢复至关重要）含 2013 人工验证实例，跨 ~15 应用 50 任务域评测 8 个模型家族：最佳 exact-match 仅 65.1%/6.7%，推断动作类型比定位更难（click F1=0.96 vs drag F1=0.76）填补了 GUI grounding 与终稀任务成功之间缺失的诊断层

为什么值得关注

computer-use agent 的年级诊断基准：最强模型仅 65% 能正确排序 GUI 转换，说明状态验证仍是短板

关键信息

论文标题：Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions?
作者：Abhishek Pillai, Samir Kumar Nayak, Yuan Chen
arXiv：https://arxiv.org/abs/2607.26041
发布时间：2026-07-28
arXiv 分类：cs.AI, cs.CV
关联标签：computer-use, benchmark, gui, desktop-agent, state-verification

English Abstract

Computer-use agents (CUAs) increasingly act through desktop GUIs to complete long-horizon tasks. Current benchmarks primarily measure end-task success or single-frame grounding. Neither isolates whether a model can reconstruct the causal, task-relevant transition produced by an action- crucial for rejecting stale observations, verifying progress, and recovering from failure. This is difficult because inference, remote input, app rendering, and screenshot capture are asynchronous: the next observation may be delayed, occluded, transient, or unrelated, then misread as progress and carried into subsequent planning. We introduce Desktop-Delta Bench (DDB), an offline step-level benchmark with 2,013 human-verified instances from novel, multi-app Linux trajectories across ~15 applications and 50 task domains. DDB trajectories targets 3 failure dimensions- state verification, source tracking, and context-aware control- through 2 complementary tasks: 463 3-frame temporal-ordering instances, including 105 with a cross-trajectory decoy, and 1,550 before-after pairs labeled from 5 actions + its payload. We evaluate 8 closed and open-source model families across 32 ordering and 16 single-action settings, observing consistent gaps. Ordering remains unsaturated: best non-decoy and decoy exact-match rates are 65.1% and 65.7%. Task context improves decoy identification by 6.9 percentage points but reduces non-decoy exact match by 2.2 points; error analysis reveals systematic copying of the presented A-B-C order. Single-action results show that inferring the action family is harder than locating it: click F1 is 0.96 vs, 0.76 for drag, while recognized drags are generally localized well. DDB, thus, complements end-to-end benchmarks by filling the missing diagnostic layer between GUI grounding and final task success, enabling targeted improvements to desktop CUA verification, reliability, and recovery.

English Summary

Desktop-Delta Bench (DDB) is an offline step-level benchmark for computer-use agents (CUAs) measuring whether models can reconstruct the causal, task-relevant transition produced by an action, crucial for rejecting stale observations, verifying progress, and recovering from failure. Contains 2,013 human-verified instances from novel multi-app Linux trajectories across ~15 applications and 50 task domains. Targets 3 failure dimensions (state verification, source tracking, context-aware control) through 463 temporal-ordering instances and 1,550 before-after pairs. Evaluating 8 model families across 32 ordering and 16 single-action settings shows consistent gaps: best exact-match rates are 65.1% (non-decoy) and 65.7% (decoy); inferring action family is harder than locating it (click F1=0.96 vs drag F1=0.76).

Obsidian Notes

内容由 opencli arxiv paper 拉取 arXiv 元数据与摘要生成。
中文导读与价值判断均锚定在条目已有摘要、论文摘要、作者、日期与分类信息上；未补充论文摘要之外的实验细节。