Measuring LLMs' ability to develop exploits
抓取时间：2026-06-24
源站：Anthropic
主题：exploit-evals
English Original
Measuring LLMs' ability to develop exploits
Authors: Newton Cheng, Keane Lucas, Winnie Xiao, Nicholas Carlini, and Milad Nasr
Published: May 22, 2026
Source: https://www.anthropic.com/research/exploit-evals
Introduction
Claude Mythos Preview's ability to develop exploits is a step-change over previous frontier models. This was one of our primary motivations for rolling out the model carefully through Project Glasswing rather than through a general release. Mythos Preview is capable of finding complex vulnerabilities, but what concerned us most in our internal testing was that Mythos Preview could both turn vulnerabilities into exploit primitives, and combine those primitives together into complete end-to-end attack chains.
When we published our Mythos Preview results, we measured its capabilities by having it search for novel zero-days and then build exploits for them. Qualitative evaluations like this are helpful for showcasing a model's capabilities—but ideally, we would have high-quality quantitative benchmarks that let us measure them precisely. The problem we faced at the time we released Mythos Preview was that no existing public exploit benchmarks were difficult enough to capture Mythos Preview's capabilities in our initial testing.
Over the last month, however, we have seen the development of two new, more challenging academic benchmarks: ExploitBench and ExploitGym. We collaborated with the researchers who produced these benchmarks to measure Mythos Preview's performance, and also ran Mythos Preview on an updated version of SCONE-bench, a benchmark we developed in collaboration with MATS and the Anthropic Fellows Program to measure smart contract exploitation. On all three benchmarks, we've found that Mythos Preview consistently outperforms all other evaluated models. We believe this is further evidence that the knowledge and expertise required to develop exploits will drop significantly as Mythos-level capabilities become more widely available.
ExploitBench: V8 bugs
ExploitBench is a benchmark to study the exploit development capabilities of large language models. It's built by Seunghyun Lee and Prof. David Brumley from Carnegie Mellon University and Bugcrowd. What makes this benchmark interesting is that it focuses on measuring the ability of language models to write complete end-to-end exploits. Prior benchmarks typically focused on measuring the ability of language models to write a "proof-of-concept" that shows the existence of a vulnerability. But a proof-of-concept only indicates that a bug is reproducible or reachable, not that an attacker could use it to actually cause harm. In ExploitBench, language models must build exploit primitives out of the vulnerability in order to enable new capabilities, such as granting the attacker arbitrary code execution (ACE).
ExploitBench decomposes the exploit development process into 16 distinct capabilities. Each of these is verified programmatically, which allows fine-grained analysis of the different intermediate capabilities required to build working exploits. The 16 capabilities are divided into five capability tiers, forming a capability ladder:
- T5 Coverage (reaching the vulnerable code path);
- T4 Reproduction (constructing a proof-of-concept to trigger the bug);
- T3 Target primitives (creating primitives confined to the V8 sandbox);
- T2 Generic primitives (breaking the sandbox to get read/write or infoleaks across the process);
- T1 Full Control (hijacking control flow or getting arbitrary code execution).
Using this framework, the authors build a V8 benchmark, which uses a set of 41 (now patched) vulnerabilities in the V8 JavaScript and WebAssembly engine that are sourced from the V8 Exploit Tracker. The V8 engine is widely used infrastructure, powering Chromium-derived applications (e.g., Chrome, Edge, Android WebView), Node.js environments (server backends), and Electron apps (e.g., VS Code, Slack, Discord). A key element of this framework is testing against security defenses: the V8 sandbox walls off the memory where a webpage's JavaScript objects live, so that a V8 bug doesn't become a foothold deeper into the browser. The highest scoring tier means arbitrary code execution in the entire V8 process (in a browser, this is like taking control over an entire tab).
Given a vulnerable build of the V8 engine and the patch that fixes a given vulnerability, the language model is instructed to build an exploit for that bug. The exploits are then scored automatically against all 16 capabilities, with no human or LLM judge. Lower tiers are checked by differential execution against the patched build; higher tiers use challenge-response functions built into V8 that are replayed across multiple randomized heap layouts, so hardcoding a leaked address won't pass. A separate static scan of the transcripts flags other forms of cheating as a backstop.
All models run on an identical ExploitBench harness with a 300 turn budget, which itself has two variants: Baseline and Nudged. In the Nudged variant, additional prompts are adaptively injected by the harness to warn the model to wrap up when close to the budget limit, or to encourage the model to use up its turn budget if it stops too early. Each variant is run for three trials. Anthropic ran all Claude models, and then provided all results and transcripts to the benchmark authors, who verified the results.
Consistent with our previous findings on Mozilla Firefox, all language models can reach or trigger the given vulnerabilities, but only models since Claude Opus 4.6 make any progress in developing primitives inside the V8 sandbox. Escaping the V8 sandbox, going from T3 to T2, is the next capability cliff; Mythos Preview is the only tested model that can reliably do so, which it does in over half the tested environments. It also achieves control flow hijack (T1) in almost half the environments in the Baseline variant. Combining Baseline and Nudged variants, Mythos Preview achieves ACE on 21 out of 41 CVEs, whereas no other model achieved even 1 ACE in either variant. The only other model to achieve ACE on the scoreboard did so in 2 out of 41 CVEs, and only using a proprietary scaffold.
In addition, the authors do a deep analysis of a few of Mythos Preview's exploit attempts. In one case, Mythos Preview was able to create a near-deterministic exploit for a bug, CVE-2023-6702, where publicly known exploits were probabilistic and uncontrolled. Because deployment of exploits may be limited to just one attempt, stability is often critical to real-world exploits that are bought and sold. How Mythos Preview achieved this was impressive as well. Seunghyun Lee, one of the authors of ExploitBench, wrote, "I have privately discussed the possibility of precisely this exploit plan with the original author of the 1-day v8CTF exploit, which we quickly dismissed due to the complexity of the approach. Mythos executed this cleanly and flawlessly without any publicly available information on this specific exploit technique."
Read more of this qualitative analysis here, and see the benchmark website at exploitbench.ai or preprint for more information.
ExploitGym
ExploitGym is a second benchmark that aims to measure language model exploitation capabilities across a broad target set. It was developed as a collaboration between UC Berkeley, the Max Planck Institute for Security and Privacy, UC Santa Barbara, and Arizona State University (with contributions from security researchers at Anthropic, OpenAI, and Google), as a follow-on to the CyberGym vulnerability-reproduction benchmark.
The authors of ExploitGym apply their evaluation framework to 898 now-patched vulnerabilities across many projects in OSS-Fuzz, the V8 engine, and the Linux kernel. Together, these three target classes cover large fractions of the world's most used software.
For a given vulnerability, the language model is provided with build information (vulnerable source code and build scripts), vulnerability information (proof-of-vulnerability; vulnerability description), runtime information (compiled binary; launch script), and a remote target running the vulnerable entrypoint. The language model is then tasked with developing a working exploit that achieves unauthorized code execution against the target, running code at a privilege level that the target's security model should make unreachable. It must then use that elevated privilege to retrieve a dynamically generated flag. An attempt is marked successful only if both the correct flag is submitted and a model judge determines the attempt to have exploited the intended vulnerability (as opposed to a different, possibly more easily exploitable, vulnerability). The evaluation framework supports toggleable security mitigations, such as the V8 heap sandbox and Linux Kernel Address Space Layout Randomization (KASLR).
The baseline framework for evaluation uses a two hour wall-clock time limit, with security mitigations toggled off, and models are run with their developers' recommended harness, e.g. Claude models are run with the Claude Code harness. All models are run with identical prompts. Anthropic ran the Opus 4.6 and Mythos Preview trials.
Within the two-hour window, Mythos Preview successfully achieves unauthorized code execution using the intended vulnerability on 157 tasks, expanding to 226 successful flag captures when including attempts involving paths to code execution that do not use the intended vulnerability. Previous generations of Claude models succeed at a significantly lower rate; for example, Opus 4.6 only achieves 15 successes with the intended vulnerability, expanding to 36 when including success via alternative vulnerability. Looking at the distribution of successes among the three classes of targets, Mythos Preview's improvements are present across all classes, and it is one of only two reported models able to frequently develop kernel exploits.
See the authors' blog or preprint for more details.
SCONE: Smart Contract Exploitation
Last year, in collaboration with MATS and the Anthropic Fellows Program, we developed the Smart Contract Exploitation benchmark (SCONE-bench) to study the ability of LLMs to find and exploit vulnerabilities in smart contracts. For each smart contract, the language model is instructed to identify a vulnerability and create an exploit to steal funds managed by the contract in local simulation. Performance is measured by the total (simulated) revenue from successful exploitations.
We ran an updated version of the benchmark that uses 12 exploits reported after the latest knowledge cutoff dates of all models (January 1, 2026), with problems sourced from the DefiHackLabs dataset. For each smart contract that was successfully exploited by the language model, we calculate the exploit's dollar value by converting the model's revenue in the native token to USD using the historical exchange rate from the day the real exploit occurred, as reported by the CoinGecko API. We then sum up the total value across all exploits, and plot this on the log-scaled figure below.
We find that Mythos Preview can exploit $35 million worth of smart contracts on this benchmark, $15 million or about 75% more than the next-closest model we tested. The latest frontier models are both able to more consistently exploit vulnerabilities (corresponding to higher attack success rates), and are able to more efficiently leverage a given exploit to steal more funds. The gap in revenue between Mythos Preview and other models is driven largely by Mythos Preview being the only model to successfully exploit every vulnerability tested. Opus 4.7 is the only other model able to exploit truebit; no other models were capable of exploiting makina in an 8-trials setting. We noted in our original post that, measured according to total revenue vs. time-of-release, the performance of models prior to Opus 4.5 follows a log-linear trajectory, with a mean doubling time of 1.1 months. Our models since Opus 4.5 continue to follow this trend, but at a doubling time of only 0.7 months. We remarked in that post that "we expect the doubling trend to plateau eventually"—but evidently we have not yet reached this plateau.
Alongside this post, we are also open-sourcing the harness and dataset for SCONE-bench here.
Conclusion
Whereas the strongest models from February of this year could only barely develop exploits in simulated scenarios with most defense measures disabled, Mythos Preview is able to construct full end-to-end exploits on the world's most widely-used software. We believe that Mythos-level models will become widely available in the next 6-12 months. As they do, this kind of exploit development will require dramatically less specialist expertise, becoming increasingly commoditized.
As models continue to become more capable, the cost of misjudging what they can do rises with it. Meeting this challenge requires building precise and comprehensive profiles of a model's capabilities, which in turn requires the development of high-quality, publicly-available benchmarks—realistic and difficult tasks built by people with deep domain expertise. The field needs more work like ExploitBench and ExploitGym, across more vulnerability classes, more targets, and more stages of the cyber attack chain. As part of our commitment to studying and mitigating the risks posed by increasingly powerful models, we are supporting the development of high-quality, rigorous evaluations of models in the cyber domain. Please reach out via our External Researcher Access Program for more details.
Better measurement is necessary but not sufficient for responsible deployment. In addition to supporting cyber defenders with Project Glasswing, we've introduced the Cyber Verification Program, allowing us to more aggressively block potentially malicious cyber threats without cutting off defenders who are using Claude to secure their own software and infrastructure.
中文翻译
衡量大语言模型开发漏洞利用（exploit）的能力
作者：Newton Cheng、Keane Lucas、Winnie Xiao、Nicholas Carlini、Milad Nasr
发布时间：2026 年 5 月 22 日
原文链接：https://www.anthropic.com/research/exploit-evals
引言
Claude Mythos Preview 在开发漏洞利用方面的能力相较此前的所有前沿模型都是一个"阶跃式"的飞跃。这也是我们选择通过 Project Glasswing 谨慎推出该模型、而非直接公开发布的主要动机之一。Mythos Preview 能够发现复杂漏洞，而在我们的内部测试中，最令我们担忧的是：它不仅能将单个漏洞转化为可用的 exploit 原语（exploit primitive），还能把这些原语组合成完整的端到端攻击链。
在最初发布 Mythos Preview 结果时，我们通过让它搜索新型 0-day 漏洞并构建漏洞利用来衡量其能力。这类定性评估能很好地展示模型能力，但更理想的做法是拥有高质量的量化基准（benchmark），让我们能精确测量这些能力。当时我们面临的问题是：现有公开的漏洞利用基准难度都不足以在我们初次测试中捕捉到 Mythos Preview 的能力上限。
但在过去一个月里，我们见证了两项新的、更具挑战性的学术基准的诞生：ExploitBench 和 ExploitGym。我们与这些基准的研究者合作，对 Mythos Preview 进行了测量；同时，我们也在 SCONE-bench（一个我们与 MATS 及 Anthropic Fellows 项目合作开发、用于衡量智能合约漏洞利用能力的基准）的更新版本上运行了 Mythos Preview。在这三项基准上，Mythos Preview 都稳定优于所有其他被测模型。我们认为，这是又一个证据：随着 Mythos 级别的能力被更广泛地获得，开发漏洞利用所需的知识与专业门槛将显著降低。
ExploitBench：V8 漏洞
ExploitBench 是一个研究大语言模型漏洞利用开发能力的基准，由卡内基梅隆大学与 Bugcrowd 的 Seunghyun Lee 和 David Brumley 教授构建。它最有意思的地方在于：它聚焦于衡量语言模型编写"完整端到端漏洞利用"的能力。此前的基准通常只关注让模型写出能证明漏洞存在的"POC（proof-of-concept）"——但 POC 只能说明漏洞可复现、可达，却无法证明攻击者可以借此真正造成危害。在 ExploitBench 中，模型必须基于漏洞构建出 exploit 原语，从而解锁新能力——例如让攻击者获得任意代码执行（ACE，Arbitrary Code Execution）。
ExploitBench 把漏洞利用开发过程拆解为 16 项独立能力，每一项都以程序化方式自动校验，从而可以对构建可用 exploit 所需的各中间能力进行细粒度分析。这 16 项能力又分为五个能力层级，构成一个"能力阶梯"：
- T5 覆盖度（reach vulnerable code path）：抵达有漏洞的代码路径；
- T4 复现（build a PoC）：构造 PoC 触发漏洞；
- T3 目标原语（sandbox-confined primitives）：构造局限于 V8 沙箱内的原语；
- T2 通用原语（escape the sandbox）：突破沙箱、获得跨进程的读写或信息泄露；
- T1 完全控制（control flow hijack / ACE）：劫持控制流或实现任意代码执行。
基于这套框架，作者们构建了一个 V8 基准，使用了 V8 Exploit Tracker 中的 41 个（已被修复的）V8 JavaScript/WebAssembly 引擎漏洞。V8 引擎是应用极广的基础设施，支撑着 Chromium 系应用（Chrome、Edge、Android WebView）、Node.js 服务端环境以及 Electron 应用（VS Code、Slack、Discord 等）。该框架的一个关键要素是引入防御测试：V8 沙箱把网页 JS 对象所在内存隔离起来，避免一个 V8 漏洞演变成对浏览器更深层的立足点。最高得分层级意味着在整个 V8 进程中获得任意代码执行（在浏览器中，相当于控制整张标签页）。
给定一个含漏洞的 V8 构建版本以及修复该漏洞的补丁，模型被要求针对该漏洞构建 exploit。评分完全自动进行，覆盖全部 16 项能力，不依赖人工或 LLM 评判。较低层级通过"对照修复版"的差分执行来校验；较高层级使用 V8 内置的"挑战-响应"函数在多种随机化的堆布局中重放——这样硬编码一个泄露地址是过不了的。此外，还会对 transcript 做静态扫描，作为兜底防止其他形式的"作弊"。
所有模型在相同的 ExploitBench harness 上运行，预算为 300 轮。基准本身分两个变体：Baseline 与 Nudged。在 Nudged 变体中，harness 会自适应地注入额外提示，例如在接近预算上限时提醒模型收尾，或在模型过早停下时鼓励它继续用完预算。每个变体跑三组试验。Anthropic 运行了所有 Claude 模型，并将所有结果与 transcript 交给基准作者复核验证。
与我们先前在 Mozilla Firefox 上的发现一致：所有语言模型都能到达或触发给定漏洞，但只有 Claude Opus 4.6 之后的模型才在 V8 沙箱内构造原语方面取得进展。下一个"能力悬崖"是从 T3 突破到 T2——逃出 V8 沙箱；Mythos Preview 是唯一能可靠做到这一点的被测模型，在超过一半的测试环境里成功。它还在 Baseline 变体中近一半的环境里达到了 T1（控制流劫持）。合并 Baseline 与 Nudged，Mythos Preview 在 41 个 CVE 上有 21 个达成 ACE，而其他模型在任一变体中都没有达到过 1 个 ACE。榜单上唯一另一个取得 ACE 的模型也仅在 2/41 个 CVE 上达成，且依赖专有的脚手架。
作者们还深度分析了 Mythos Preview 的几次漏洞利用尝试。其中一例是，Mythos Preview 针对 CVE-2023-6702 构建出了一个近乎确定性的 exploit，而公开已知的 exploit 都是概率性、不可控的。由于真实 exploit 的部署往往只能尝试一次，稳定性通常对现实中被买卖的 exploit 至关重要。Mythos Preview 实现这一点的方式也令人印象深刻。ExploitBench 作者 Seunghyun Lee 写道："我曾私下与 1-day v8CTF exploit 的原作者讨论过这种 exploit 计划的可行性，因为复杂度太高我们很快放弃了。Mythos 在没有这条特定 exploit 技术的任何公开信息的情况下，干净利落、毫无瑕疵地执行了它。"
更多定性分析请见这里，基准站点见 exploitbench.ai，或参阅其预印本。
ExploitGym
ExploitGym 是另一项基准，旨在跨更广泛的目标集合衡量语言模型的漏洞利用能力。它由 UC Berkeley、Max Planck Institute for Security and Privacy、UC Santa Barbara 与 Arizona State University 合作开发（Anthropic、OpenAI 与 Google 的安全研究者亦有贡献），是 CyberGym 漏洞复现基准的延续。
ExploitGym 的作者把这一评测框架应用到了 OSS-Fuzz、V8 引擎与 Linux 内核中共 898 个已被修复的漏洞上。这三类目标合计覆盖了世界上使用最广软件中的相当大一部分。
针对每一个漏洞，模型会获得：构建信息（漏洞源码与构建脚本）、漏洞信息（漏洞证明与描述）、运行信息（已编译的二进制与启动脚本）以及一个运行着该漏洞入口点的远程目标。模型的任务是开发一个能针对该目标取得未授权代码执行的漏洞利用——所执行代码的权限等级应高于目标安全模型所应允许的等级。然后它必须借助这一被提升的权限取回一个动态生成的 flag。只有当 flag 提交正确、且一个模型评审判定该尝试利用的是目标漏洞（而非另一个可能更易利用的漏洞）时，该次尝试才算成功。评测框架支持可切换的安全缓解措施，如 V8 堆沙箱与 Linux 内核地址空间布局随机化（KASLR）。
基线评测框架以两小时墙钟时间为上限，关闭安全缓解措施；模型以各自开发者推荐的 harness 运行（如 Claude 模型使用 Claude Code harness）。所有模型使用相同的提示词。Anthropic 运行了 Opus 4.6 与 Mythos Preview 的试验。
在两小时窗口内，Mythos Preview 成功在 157 个任务上用目标漏洞实现未授权代码执行；如果把那些没有利用目标漏洞、但同样拿到 flag 的尝试也算进来，则扩展到 226 次成功。上一代 Claude 模型的成功率明显更低：例如 Opus 4.6 仅在 15 个任务上以目标漏洞成功，纳入利用其他漏洞的"附加成功"也只有 36 次。观察三类目标的成功分布，Mythos Preview 的提升在各类目标上都存在，并且它是仅有的两个能频繁开发内核 exploit 的模型之一。
SCONE：智能合约漏洞利用
去年，我们与 MATS 和 Anthropic Fellows 项目合作开发了智能合约漏洞利用基准（SCONE-bench），以研究大语言模型在智能合约中发现并利用漏洞的能力。针对每一份智能合约，模型被要求识别漏洞，并在本地模拟中创建一个漏洞利用，偷走合约管理的资金。性能以成功 exploit 的总（模拟）收益衡量。
我们跑了更新版基准，使用了 12 个在所有模型最新知识截止日期（2026 年 1 月 1 日）之后被报告的漏洞，题目来自 DefiHackLabs 数据集)。对于模型成功 exploit 的每份智能合约，我们用真实 exploit 发生当天的历史汇率（来自 CoinGecko API）把模型收益换算成美元。然后对所有 exploit 的金额求和，并绘制到下面这张对数坐标图上。
我们发现 Mythos Preview 在该基准上能 exploit 出 3500 万美元价值的智能合约，比第二名高出约 1500 万美元（高出约 75%）。最新一代前沿模型既能在更多漏洞上一致地取得 exploit（攻击成功率更高），也能更高效地利用单个漏洞偷到更多资金。Mythos Preview 与其他模型之间的金额差距，很大程度上来自：它是唯一一个对所测每个漏洞都成功 exploit 的模型。Opus 4.7 是另一个唯一能 exploit truebit 的模型；在 8 次试验设置下，没有任何其他模型能 exploit makina。在我们原帖中我们曾指出：以"总收益 vs 发布时间"衡量，Opus 4.5 之前模型的性能呈对数线性增长，平均翻倍时间 1.1 个月。Opus 4.5 之后的模型沿袭了这一趋势，但翻倍时间缩短到 0.7 个月。我们当时曾评论"我们预期这一翻倍趋势终将趋于平稳"——但显然，我们尚未到达那个平台期。
随本帖发布的还有我们开源的 SCONE-bench harness 与数据集，见这里。
结论
如果说今年 2 月最强的模型只能在大多数防御措施都被关闭的模拟场景中"勉强"开发 exploit，那么 Mythos Preview 已经在世界上最广泛使用的软件上构建出完整的端到端漏洞利用。我们认为 Mythos 级别的模型将在未来 6-12 个月内被广泛获得。一旦如此，这类漏洞利用开发所需的专业门槛将大幅降低，并被日益"商品化"。
随着模型能力持续增强，误判其能力边界的代价也随之上升。应对这一挑战要求我们构建精确而全面的能力画像，而这就要求开发高质量的、公开可用的基准——由具有深厚领域专长的人构建的、真实而困难的任务。整个领域需要更多像 ExploitBench 和 ExploitGym 这样的工作，覆盖更多漏洞类别、更多目标、更多网络攻击链环节。作为我们研究与缓解日益强大模型风险承诺的一部分，我们将持续支持网络域内高质量、严谨的模型评测工作。欢迎通过我们的外部研究者访问计划联系以了解详情。
更精准的测量是负责任部署的必要而非充分条件。除了通过 Project Glasswing 支持网络防御方，我们还引入了网络验证项目（Cyber Verification Program），使我们能够在不切断那些用 Claude 来保护自己软件与基础设施的防御方的前提下，更积极地拦截潜在恶意网络威胁。
*本文由 openclaw cron 自动抓取/汇总生成。*