CUA-Universe: Scalable Hybrid GUI+CLI Environment for Computer-Use Agents
Source: https://arxiv.org/abs/2609.05374
Authors: Haoting Shi, Wenhao Wang, Weicheng Fang, Yaozhong Liang, Tian Jin, Pengxiang Zhao, Guangyi Liu, Siheng Chen, Yanfeng Wang
Published: 2026-09-04
Categories: cs.AI
PDF: https://arxiv.org/pdf/2609.05374v1
Abstract
Computer-use agents have advanced on benchmarks like OSWorld and AndroidWorld, but still act mostly through the GUI, often producing inefficient trajectories. Real-world computer work is hybrid, combining visual-state inspection with precise, high-throughput command-line operations, so capable agents must coordinate both modalities over shared application state. Yet scalable hybrid environments remain scarce because supporting both GUI and CLI over real applications typically requires substantial manual engineering for each application. Existing agents also struggle to use the two interfaces complementarily: CLI-native agents lack visual perception for tasks involving interface state or layout, while GUI-native agents are inefficient for operations better executed through commands. We introduce CUA-Universe, a scalable environment-to-data pipeline that turns real desktop software into hybrid GUI+CLI environments. App-Forge adapts applications into reproducible VMs and command-line surfaces it discovers, wraps, or generates, scaling to 16 applications; Task-Weave synthesizes diverse hybrid tasks of controllable difficulty from reusable operations over seed files; and Path-Steer steers rollouts along efficient hybrid paths and harvests verified trajectories for post-training. Training on this data shifts behavior from inefficient GUI interaction and brittle CLI scripting toward effective GUI+CLI orchestration. Our 9B model improves both success and efficiency on CUA-Verse (Score +39.3 pts; -37% steps, -60% tokens), OSWorld (SR +16.8 pts; -57% steps, -44% tokens), and OSWorld-MCP (Score +7.84 pts; -27% steps, -30% tokens). CUA-Universe provides a scalable path toward more capable and efficient computer-use agents.
Key Findings
- Pipeline (3 stages): App-Forge adapts 16 desktop apps into reproducible VMs + discovered/wrapped/generated CLIs; Task-Weave synthesizes controllable-difficulty hybrid tasks from seed operations; Path-Steer steers rollouts along efficient hybrid paths and harvests verified trajectories for post-training.
- Model gain: A 9B model trained on this data improves on CUA-Verse (Score +39.3; steps −37%, tokens −60%), OSWorld (SR +16.8; steps −57%, tokens −44%), and OSWorld-MCP (Score +7.84; steps −27%, tokens −30%).
- Why it matters: Real computer work is hybrid — GUI for visual state inspection, CLI for precise high-throughput operations. Prior agents were GUI-heavy or CLI-only; CUA-Universe gives both a shared scalable training substrate.
- Scope: 21 pages, 9 figures.
中文概要
本文提出 CUA-Universe，一个可扩展的 hybrid GUI+CLI 智能体环境生产线。原型系统在 OSWorld 等基准上大多仅靠 GUI，实际电脑作业却是视觉与命令行混用。App-Forge 自动把 16 款桌面软件适配为可复现的虚机并暴露 CLI 接口；Task-Weave 从种子文件上编织可控难度的混合任务；Path-Steer 带着全程走高效轨迹并收集验证后的轨迹用于微调。全补齐后的 9B 模型在 CUA-Verse 上 Score +39.3、步数 −37%、token −60%；OSWorld SR +16.8；OSWorld-MCP Score +7.84。证明“多模态协同”比单路线 GUI 更准且更省 token。