JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition
- ID: 71307c33
- 原文链接: https://arxiv.org/abs/2609.10451
- PDF: https://arxiv.org/pdf/2609.10451v1
- 作者: Zixiang Chen, Yuheng Lu, Zihao Cheng, Zeming Liu, Jizeng Bai, Ziye Huang, Zhiyin Lin, Zihan Li, Yuhang Guo, Yunhong Wang, Haifeng Wang
- 日期: 2026-09-09
- 更新: 2026-09-09
- 分类: agents
- 来源类型: paper
- 标签: gui-agents, benchmark, cross-device, android, windows, ubuntu, field-note
- 质量评分: 4/5
- 抓取时间: 2026-09-11T04:23:41
中文导读
GUI 智能体评测的盲区补上了：任务=轻量类型系统下的输入-输出变换，自动组合跨 Android/Windows/Ubuntu 工作流；SOTA 开源模型在状态转移跨平台推理长程依赖上集体失守
为什么值得关注
GUI 智能体评测的盲区补上了：任务=轻量类型系统下的输入-输出变换，自动组合跨 Android/Windows/Ubuntu 工作流；SOTA 开源模型在状态转移跨平台推理长程依赖上集体失守
关键信息
- 论文标题：JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition
- 作者：Zixiang Chen, Yuheng Lu, Zihao Cheng, Zeming Liu, Jizeng Bai, Ziye Huang, Zhiyin Lin, Zihan Li, Yuhang Guo, Yunhong Wang, Haifeng Wang
- arXiv：https://arxiv.org/abs/2609.10451
- 发布时间：2026-09-09
- arXiv 分类：cs.AI
- 关联标签：gui-agents, benchmark, cross-device, android, windows, ubuntu, field-note
English Abstract
Real-world GUI usage frequently involves workflows that span multiple devices and platforms, requiring the transfer of intermediate results, maintenance of shared state, and coordination across heterogeneous environments. However, existing GUI benchmarks overwhelmingly evaluate agents on single-device, statically defined tasks, thus leaving such cross-device capabilities largely unexamined, resulting in an overly optimistic assessment of agents' readiness for real-world usage. We introduce JarvisGUI, a dynamic benchmark that evaluates GUI agents on cross-device workflows requiring coordinated interaction across heterogeneous platforms, including Android, Windows, and Ubuntu. Specifically, JarvisGUI formulates GUI tasks as input-output transformations under a lightweight type system, which allows us to automatically compose multi-step, cross-device workflows and dynamically evaluate agent performance within a unified framework. By evaluating agents in virtual environments spanning multiple operating systems, JarvisGUI reveals that state-of-the-art open-source GUI agents struggle with the state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management required for real-world workflows, exposing a critical capability gap invisible to existing benchmarks.
English Summary
JarvisGUI is a dynamic benchmark evaluating GUI agents on cross-device workflows requiring coordinated interaction across heterogeneous platforms including Android, Windows, and Ubuntu. It formulates GUI tasks as input-output transformations under a lightweight type system, enabling automatic composition of multi-step cross-device workflows and dynamic evaluation in a unified framework. Across virtual environments spanning multiple operating systems, state-of-the-art open-source GUI agents struggle with state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management, exposing a capability gap invisible to existing single-device benchmarks.
Obsidian Notes
- 内容由
opencli arxiv paper拉取 arXiv 元数据与摘要生成。 - 中文导读与价值判断均锚定在条目已有摘要、论文摘要、作者、日期与分类信息上；未补充论文摘要之外的实验细节。