MacArena: Benchmarking Computer Use Agents on an Online macOS Environment
- ID: 2a6b9df8
- Source: arXiv
- Original URL: https://arxiv.org/abs/2606.06560
- PDF: https://arxiv.org/pdf/2606.06560v1
- Authors: Victor Muryn, Maksym Shamrai, Sofiia Mazepa, Yehor Khodysko
- Published: 2026-06-04
- Categories: cs.LG, cs.AI, cs.HC
- Comments: Accepted to the Second Workshop on Agents in the Wild: Safety, Security, and Beyond (AIWILD) at ICML 2026
- AAIF category: agents
- Tags: computer-use-agents, macos, benchmark, gui-agents, arxiv, cs.lg
- Quality score: 4/5
- Fetched at: 2026-08-03T12:20:41+08:00
One-liner
MacArena 证明 GUI 智能体的跨平台能力不能由 Linux/OSWorld 成绩直接外推
Chinese summary
MacArena 将 CUA 评估扩展到 Apple Silicon 上的在线 macOS 环境，包含 50 个应用421 个人工验证任务论文强调 macOS-native 任务会让既有模型排序反转，领先模型在 MacArena 子集上落后超过 26%，提示 GUI 智能体需要跨平台而非只熟悉 Linux/OSWorld 分布
English summary
Computer-use agents (CUAs) operate graphical user interfaces (GUIs) through vision and control primitives, and their capabilities have advanced rapidly, driven in part by standardized online evaluation benchmarks such as OSWorld, which serve both as evaluation tools and as training environments for reinforcement learning. However, macOS remains underserved in this landscape: the only existing benchmark, macOSWorld, covers a narrow slice of first-party applications with simpler tasks, and runs on x86 virtual machines incompatible with Apple Silicon. We introduce MacArena, a benchmark of 421 manually verified tasks spanning 50 applications that combines a curated port of OSWorld tasks, content sourced from macOSWorld, and 49 new macOS-native tasks, all running on Apple's native Virtualization framework on Apple Silicon....
arXiv abstract
Computer-use agents (CUAs) operate graphical user interfaces (GUIs) through vision and control primitives, and their capabilities have advanced rapidly, driven in part by standardized online evaluation benchmarks such as OSWorld, which serve both as evaluation tools and as training environments for reinforcement learning. However, macOS remains underserved in this landscape: the only existing benchmark, macOSWorld, covers a narrow slice of first-party applications with simpler tasks, and runs on x86 virtual machines incompatible with Apple Silicon. We introduce MacArena, a benchmark of 421 manually verified tasks spanning 50 applications that combines a curated port of OSWorld tasks, content sourced from macOSWorld, and 49 new macOS-native tasks, all running on Apple's native Virtualization framework on Apple Silicon. We argue that macOS presents distinct GUI challenges beyond what Linux-based benchmarks capture, and our evaluation supports this claim: strong model performance on existing benchmarks can reflect familiarity with task distributions rather than genuine cross-platform GUI competence. Notably, model rankings invert between ported and macOS-native tasks, with a leading model trailing by over 26% on the MacArena subset, suggesting that macOS poses a genuinely harder environment for current GUI agents.
Why it matters for AAIF
- Grounded from arXiv metadata fetched with
opencli arxiv paperand the existing AAIF entry summary. - This backfill turns an entry-only card into a readable local content page without changing
entries.json.