Sample More, Reflect Less: 重复采样在等令牌成本下击败自反思方法（1.5B-7B）

Sample More, Reflect Less: 重复采样在等令牌成本下击败自反思方法（1.5B-7B）

Source: https://arxiv.org/abs/2607.28576

Content fetched: 2026-08-02T12:19:09+08:00

Grounding: opencli arxiv paper

一句话

实验表明，在等令牌成本下，重复采样击败了 Self-Refine/Reflexion 等自反思方法，18 项自检查对比全为负面

关键信息

• arXiv ID: 2607.28576
• English title: Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B
• Authors: Iliya Mirzaei
• Submitted/Published: 2026-07-30
• Updated: 2026-07-30
• Subjects: cs.CL, cs.AI, cs.LG
• PDF: https://arxiv.org/pdf/2607.28576v1
• Tags: self-refine, repeated-sampling, token-cost, reasoning, evaluation
• Quality score: 4

中文摘要

本文严格测试了自反思方法（Self-RefineReflexion自辩论Best-of-N 选择）是否真正超越了单纯多生成文本的效果实验设计：七种方法三个开源模型规模（1.5B/3B/7B）两个数学基准每个 150 题，统计所有生成 token（包括批评反思辩论轮次）结果：在等令牌成本下，没有一种方法可靠地优于重复采样；十项对比中方法明显更差，全部涉及自检查；随着模型变大，两种自检查类型的差异加剧

English Summary

This paper rigorously tests whether self-reflection methods (Self-Refine, Reflexion, self-debate, Best-of-N selection) actually help beyond simply generating more text. The author runs a designed experiment: seven methods, open models of 1.5B/3B/7B parameters, two math benchmarks, 150 questions each. Every generated token is counted including those spent on critiques, reflections, and debate turns. Results: no method is reliably better than repeated sampling at equal cost. Ten comparisons show methods are reliably worse, all involving self-inspection. The two kinds of self-inspection part company as models grow: choosing stops hurting at larger scales.

Why it matters

实验表明，在等令牌成本下，重复采样击败了 Self-Refine/Reflexion 等自反思方法，18 项自检查对比全为负面

arXiv Abstract

Methods that make a language model plan, criticise and rewrite its own answer, reflect on mistakes, pick the best of several attempts, or debate with copies of itself nearly all make it generate far more text than a single chain of thought. Because generating more text raises accuracy by itself, a gain over one chain of thought does not show the method's idea is what helped. Wang et al. (2024) reported that a simple baseline, sampling the same question repeatedly and keeping the most common answer, often wins once budgets are comparable, but gave point estimates with no confidence intervals or significance tests. We rerun that comparison as a designed experiment: seven methods, open models of 1.5B, 3B and 7B parameters, two mathematics benchmarks, 150 questions each. We count every generated token, including those spent on critiques, reflections, debate turns and checking, and compare each method against repeated sampling at its own measured cost. All 36 comparisons are paired by question, with bootstrap intervals and multiplicity correction. No method is reliably better than repeated sampling at equal cost anywhere. Ten are reliably worse, all of them methods where the model inspects its own output, and all 18 self-inspection comparisons are negative. The two kinds of self-inspection part company as models grow. Choosing stops hurting: taking Best-of-N's eight samples and just counting the most common answer beats letting the model pick by 8.0 and 11.3 points at 1.5B, but only 2.0 and 1.3 at 7B, no longer distinguishable from zero. Rewriting does not recover: Self-Refine and a forced Reflexion stay 3.6 to 10.1 points below baseline at 7B. Reflexion as published never triggered its own retry on the smallest model. It judged itself correct every time and silently became a single chain of thought. We release code, prompts, all generations, and our verification scripts.

Source Metadata

{
  "id": "2607.28576",
  "title": "Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B",
  "authors": "Iliya Mirzaei",
  "abstract": "Methods that make a language model plan, criticise and rewrite its own answer, reflect on mistakes, pick the best of several attempts, or debate with copies of itself nearly all make it generate far more text than a single chain of thought. Because generating more text raises accuracy by itself, a gain over one chain of thought does not show the method's idea is what helped. Wang et al. (2024) reported that a simple baseline, sampling the same question repeatedly and keeping the most common answer, often wins once budgets are comparable, but gave point estimates with no confidence intervals or significance tests. We rerun that comparison as a designed experiment: seven methods, open models of 1.5B, 3B and 7B parameters, two mathematics benchmarks, 150 questions each. We count every generated token, including those spent on critiques, reflections, debate turns and checking, and compare each method against repeated sampling at its own measured cost. All 36 comparisons are paired by question, with bootstrap intervals and multiplicity correction. No method is reliably better than repeated sampling at equal cost anywhere. Ten are reliably worse, all of them methods where the model inspects its own output, and all 18 self-inspection comparisons are negative. The two kinds of self-inspection part company as models grow. Choosing stops hurting: taking Best-of-N's eight samples and just counting the most common answer beats letting the model pick by 8.0 and 11.3 points at 1.5B, but only 2.0 and 1.3 at 7B, no longer distinguishable from zero. Rewriting does not recover: Self-Refine and a forced Reflexion stay 3.6 to 10.1 points below baseline at 7B. Reflexion as published never triggered its own retry on the smallest model. It judged itself correct every time and silently became a single chain of thought. We release code, prompts, all generations, and our verification scripts.",
  "published": "2026-07-30",
  "updated": "2026-07-30",
  "primary_category": "cs.CL",
  "categories": "cs.CL, cs.AI, cs.LG",
  "comment": "",
  "pdf": "https://arxiv.org/pdf/2607.28576v1",
  "url": "https://arxiv.org/abs/2607.28576"
}