FrontierHarness Eval: 9 harnesses x 12 configs on Kimi K3 same model, 17.5x cost spread

ID: 0fe437a1
原文链接: https://frontierharness.org
作者: Runta
日期: 2026-09-01
分类: agents
来源类型: article
标签: agent-harness, agent-eval, frontierharness, kimi-k3, cost-comparison
质量评分: 5/5
抓取时间: 2026-09-03T15:30:00Z

中文导读

FrontierHarness Eval v1.0 在 Kimi K3 上对比 9 个 harness（Codex / Claude Code / DSH Creator/Standard/Minimal/Pi / Hermes / OpenCode / Oh My Pi / Kimi Code / Exo）共 12 个配置，30 个任务 / 360 次冷启动同 checkpoint 评测Pass rate 跨度 50.0%66.7%（Codex 66.7 / Claude Code 63.3 / Hermes 50 / Exo 53.3）；中位 $/pass 跨度 $1.05$18.34（约 17.5 倍），摊销 $/success 跨度 $0.0615$0.2880（约 4.7 倍，且排序会翻转）CC token-weighted cache 中位 25.0%，明显低于全场 67.8%；单条案例题（python-statemachine）CC 跑到 381 turns / $64.36 / 15.7% cacheLive 榜 + Runta 博文 + repo 一并公开，v1.1 计划扩 harnessmodel 矩阵提示：实现侧分叉已从推理栈延伸到 agent 会话壳，读榜时需拆列，单模型横切不能直接当整体排名

关键信息

条目标题：FrontierHarness Eval: 9 harnesses x 12 configs on Kimi K3 same model, 17.5x cost spread
原文链接：https://frontierharness.org
发布日期：2026-09-01
作者/平台：Runta · blog
分类标签：agents · agent-harness, agent-eval, frontierharness, kimi-k3, cost-comparison

English Abstract

FrontierHarness Eval v1.0 evaluates 9 harnesses (Codex, Claude Code, DSH Creator/Standard/Minimal/Pi, Hermes, OpenCode, Oh My Pi, Kimi Code, Exo) over 12 configurations on Kimi K3 with 30 tasks and 360 cold-start identical-checkpoint runs. Pass rate spans 50.0%66.7% (Codex 66.7, Claude Code 63.3, Hermes 50, Exo 53.3). Median cost per successful task spreads $1.05$18.34 (17.5x); amortized $/success spans $0.0615$0.2880 (4.7x and reverses the ranking). Claude Code's token-weighted cache median is 25.0%, well below the field median of 67.8%; one case task (python-statemachine) reaches 381 turns and $64.36 at 15.7% cache. Live leaderboard, Runta blog post, and the eval repository are publicly released; v1.1 plans a harnessmodel matrix.

Obsidian Notes

由 AAIF daily intake (evening 2026-09-03) 通过 Obsidian 当日调研/Cubox 笔记落盘。
摘要锚定在条目原文（https://frontierharness.org）与同日 DeepResearch references 上；本文件没有补抓额外实验/正文细节。