Can Open-Source LLM Agents Replace Static Application Security Testing Tools? An Empirical Asses...

这篇实证研究比较 3 个 Ollama 托管开源模型驱动的通用 LLM agent 与 Bandit SAST 工具，在 precisionrecallfalse positive 和复合分数上评估其安全扫描表现结论是否定性的：现实条件下，现代开源 GenAI agent 还不适合替代专业 SAST 工具

Can Open-Source LLM Agents Replace Static Application Security Testing Tools? An Empirical Assessment

ID: d7fb15b9
Source: arXiv
Original URL: https://arxiv.org/abs/2606.11672
PDF: https://arxiv.org/pdf/2606.11672v1
Authors: Derek Yohn, Luke Flancher, Mirajul Islam, Khaled Slhoub
Published: 2026-06-10
Categories: cs.CR, cs.AI
Comments: Keywords: Agentic AI, Cybersecurity, Large Language Models, Static Application Security Testing, Model performance evaluation
AAIF category: coding
Tags: agent-security, sast, cybersecurity, evaluation, arxiv, cs.cr
Quality score: 4/5
Fetched at: 2026-08-03T12:20:42+08:00

One-liner

开源 LLM agent 暂不能在现实 SAST 场景中替代成熟静态安全扫描工具

Chinese summary

English summary

This paper explores the value of agentic AI tools for cybersecurity purposes. We evaluate the efficacy of a general-purpose GenAI Large Language Model- (GenAI-) based agent when powered by three different Ollama-hosted general-purpose open source models. We assess each agent's performance using precision, recall, false positive count, and a calculated composite score based upon the interplay of the captured metrics, against the baseline performance of an existing, vetted Static Application Security Testing (SAST) tool, Bandit. Our findings refute the notion that a modern open-source GenAI LLM-based agent is currently suitable for the specialized task of SAST scanning under realistic conditions.

arXiv abstract

Why it matters for AAIF

Grounded from arXiv metadata fetched with opencli arxiv paper and the existing AAIF entry summary.
This backfill turns an entry-only card into a readable local content page without changing entries.json.