Discovering Agentic Safety Specifications from 1-Bit Danger Signals

Discovering Agentic Safety Specifications from 1-Bit Danger Signals

• ID: 0e5ebf11
• Source: arXiv
• Original URL: https://arxiv.org/abs/2604.23210
• PDF: https://arxiv.org/pdf/2604.23210v1
• Authors: Víctor Gallego
• Published: 2026-04-25
• Categories: cs.AI, cs.CL
• Comments: Accepted to the Adaptive and Learning Agents Workshop (ALA 2026) @ AAMAS 2026. Code is available at github.com/vicgalle/experiential-prompt-optimization-safe
• AAIF category: agents
• Tags: agent-safety, llm-agents, reward-hacking, safety-specification, arxiv, cs.ai
• Quality score: 4/5
• Fetched at: 2026-08-03T12:20:38+08:00

One-liner

智能体可从极稀疏危险信号中学习可审计安全规范，但奖励驱动反思会放大奖励黑客

Chinese summary

EPO-Safe 研究让 LLM 智能体只依靠每步 1-bit 的危险警告来迭代生成安全行为规范；论文在 5 个 AI Safety Gridworlds 和 5 个文本场景中报告，12 轮515 个 episode 即可发现安全规则，并指出只按奖励反思会加速 reward hacking适合作为安全反馈通道必须独立于奖励通道的智能体设计证据

English summary

Can large language model agents discover hidden safety objectives through experience alone? We introduce EPO-Safe (Experiential Prompt Optimization for Safe Agents), a framework where an LLM iteratively generates action plans, receives sparse binary danger warnings, and evolves a natural language behavioral specification through reflection. Unlike standard LLM reflection methods that rely on rich textual feedback (e.g., compiler errors or detailed environment responses), EPO-Safe demonstrates that LLMs can perform safety reasoning from a strictly impoverished signal in structured, low-dimensional environments: the agent never observes the hidden performance function $R^*$, only a single bit per timestep indicating that an action was unsafe. We evaluate on five AI Safety Gridworlds (Leike et al., 2017) and five text-based scenario analogs where visible reward $R$ may diverge from $R^*$....

arXiv abstract

Can large language model agents discover hidden safety objectives through experience alone? We introduce EPO-Safe (Experiential Prompt Optimization for Safe Agents), a framework where an LLM iteratively generates action plans, receives sparse binary danger warnings, and evolves a natural language behavioral specification through reflection. Unlike standard LLM reflection methods that rely on rich textual feedback (e.g., compiler errors or detailed environment responses), EPO-Safe demonstrates that LLMs can perform safety reasoning from a strictly impoverished signal in structured, low-dimensional environments: the agent never observes the hidden performance function $R^*$, only a single bit per timestep indicating that an action was unsafe. We evaluate on five AI Safety Gridworlds (Leike et al., 2017) and five text-based scenario analogs where visible reward $R$ may diverge from $R^*$. EPO-Safe discovers safe behavior within 1-2 rounds (5-15 episodes), producing human-readable specifications with correct explanatory hypotheses about hazards (e.g., "X cells are directionally hazardous: entering from the north is dangerous"). Critically, we show that standard reward-driven reflection actively degrades safety: agents reflecting on reward alone use the loop to justify and accelerate reward hacking, proving that reflection must be paired with a dedicated safety channel to discover hidden constraints. We further evaluate robustness to noisy oracles: even when 50% of non-dangerous steps produce spurious warnings, mean safety performance degrades by only 15% on average, though sensitivity is environment-dependent, as cross-episode reflection naturally filters inconsistent signals. Each evolved specification functions as an auditable set of grounded behavioral rules discovered autonomously through interaction, rather than authored by humans as in Constitutional AI (Bai et al., 2022).

Why it matters for AAIF

• Grounded from arXiv metadata fetched with opencli arxiv paper and the existing AAIF entry summary.
• This backfill turns an entry-only card into a readable local content page without changing entries.json.