On-Policy Distillation for LLM Safety: A Routing Approach to Template-Robust Realignment
Authors: Yongjian Guo, Wanlun Ma, Lingyu Shen, Xi Xiao, Sheng Wen
Published: 2026-07-29
Categories: cs.AI, cs.CL, cs.CR, cs.LG
arXiv: 2607.27081
Abstract
Fine-tuning is the dominant paradigm for specializing large language models (LLMs), yet it exposes a critical vulnerability: malicious data providers can embed harmful behaviors into downstream corpora, creating models that retain professional skills while violating human values on demand. Existing safety-realignment defenses often fail in practice due to three key limitations: they frequently cause catastrophic forgetting of specialized skills; their effectiveness collapses when the defender cannot observe the attacker's prompt template; and successfully realigned models remain susceptible to re-jailbreaking via simple system prompt switches. We propose Routing-based On-Policy Distillation (ROPD), a novel realignment framework that models the divergence between aligned and compromised output probability distributions rather than fitting specific prompt templates. Extensive experiments comparing ROPD against four state-of-the-art baselines across three datasets and three base models demonstrate that ROPD substantially mitigates template-mismatch risks, maintaining superior robustness in both defense effectiveness and capability preservation.
中文概述
这篇论文针对微调后的恶意行为嵌入问题，提出 ROPD（路由式在策蒸馏）重新对齐框架。现有防御存在三大局限：灾难性遗忘专业能力、模板不匹配时效果崩塌、简单系统提示词切换即可重新越狱。ROPD 建模对齐与被篡改输出分布之间的散度，而非拟合特定提示词模板。实验表明 ROPD 在防御有效性和能力保留两方面均优于四个 SOTA 基线。
关键发现
- 现有安全重对齐的三大局限：灾难性遗忘、模板不匹配失效、简单重越狱
- ROPD 建模对齐与被篡改输出分布的散度，而非拟合提示词模板
- 在三个数据集、三个基座模型上优于四个 SOTA 基线
- 模板偏移下的性能降解可忽略，而现有方法严重退化
信息来源
- 论文链接: https://arxiv.org/abs/2607.27081
- PDF: https://arxiv.org/pdf/2607.27081v1
- 主要类别: cs.AI, cs.CL, cs.CR, cs.LG