Phantom Gains: Auditing Self-Improvement Against a Measured Null
- arXiv: 2608.20290
- Authors: Cheng Xu, Nan Yan, Liming Chen, M-Tahar Kechadi
- Published: 2026-08-20; categories: cs.AI, cs.CL
Abstract
Whether a language model has improved itself is increasingly judged not by mean accuracy but by which individual problems it gains and loses. Tracking these transitions means differencing two noisy estimates, leaving them vulnerable to measurement artifacts. Auditing three rounds of rank-$32$ LoRA self-training on Qwen3-8B against a frozen control pushed through the identical pipeline, we identify seven measurement failures, each of which inverts a reported finding when its control is absent. Several are standard practice. A ledger built on a single greedy decode manufactures capability changes on an untrained model, largely an artifact of inference batching; the expansion statistic separating acquisition from sharpening assigns that same model a rate of $0.280$. The natural threshold repair does not survive replication: estimated across the frozen comparisons such a design already contains, its null stays non-zero. We replace it with a per-problem exact test against a pooled baseline under false-discovery-rate control, which detects nothing on any held-out replicate and is unchanged under the multiple-testing rule, error rate and pool size. Applied to a ladder of arms matched in stream, volume and evaluation, the audit finds that external distillation improves problems the base model rarely reaches while three forms of self-training do not; a regression rejects this asymmetry as a by-product of distillation's larger overall gain ($p < 10^{-8}$). On the far smaller set of problems the base model never reaches, the evidence is inconclusive, while self-training corrupts problems solved at baseline at rates well above the measured floor. Transition-level auditing therefore requires a separately measured null for every statistic it reports: nulls that cost no new experiments, built from baseline replicates a multi-arm study already owns, though not from as few as most possess.
Why it matters (AAIF scan)
对自我改进评估的方法学审计：三轮 rank-32 LoRA 自训练 (Qwen3-8B) 与走完全相同管线的冻结对照做差分，识别出 7 种测量失效，任何一种在缺少对照时都会反转结论例如单一 greedy decode 的能力账本会把批处理推理伪影当成能力变化，给未训练模型记出 0.280 的习得率；换成逐题精确检验 + FDR 控制后，留出副本上检测不到任何自训练收益，而外部蒸馏确实改进了基座模型很少触及的问题（回归排除不对称性 p<1e-8）凡是报告 agent/模型自我改进效果的工作，先过这篇的审计清单
Source: https://arxiv.org/abs/2608.20290
Captured: 2026-08-22 (AAIF content-fetcher)