Overtraining as the path to human-like AI

Overtraining as the path to human-like AI

• source_url: https://seangoedecke.com/overtraining-as-the-path-to-human-like-ai
• source_type: article
• platform: blog
• author: Sean Goedecke
• original_date: 2026-07-20
• added_date: 2026-07-20
• local_path: OpenClaw定时任务/AK-RSS-Digest（89源精选）/2026-07-20-AK-RSS-Digest.md
• quality_score: 4

摘要（中文）

文章转述 Gwern 的 catapulting 和 grokking 思路：如果当前模型缺的是更深层泛化，下一步能力跃迁可能不只靠继续堆数据，而是让超大模型在较小数据集上长时间训练，被迫压出更简洁的规则它同时指出这种反向训练策略的代价：需要百兆亿参数级模型小数据集长训练和组织耐心，风险很高

Summary (English)

Sean Goedecke explains the argument that overtraining very large models on smaller datasets might produce grokking-like jumps toward deeper generalization, while emphasizing the technical and organizational risks.

One-liner

Overtraining 路线的争议点不在口号，而在用长训练赌更深层泛化

原文 / 元数据抓取

Overtraining as the path to human

原文链接: https://seangoedecke.com/overtraining-as-the-path-to-human-like-ai/

The anonymous blogger Gwern recently completed a thirteen thousand word post called _Human-like Neural Nets by Catapulting_, in which he offers a theory about why LLMs don’t possess truly flexible human-like intelligence, and how we might train LLMs that do. Theories like this are entirely unremarkable: every ~~crank~~ researcher on the internet has a theory about how to crack AI. But _Gwern_ is remarkable. Outside of OpenAI itself, Gwern is the earliest person to anticipate the potential of large language models, and the scaling arms-race involved in making them larger and more powerful still. I often cite Leopold Aschenbrenner’s _Situational Awareness_ as an example of someone correctly predicting the future of AI. Written in 2024, just after the release of GPT-4, Aschenbrenner gets a lot of things right: the rush to build billion or trillion-dollar GPU clusters, the importance of the code _around_ the LLM (what he calls “unhobbling”)1, and the fact that scaling would continue through the decade. Gwern’s essay _The Scaling Hypothesis_ anticipated the broad strokes _in 2020_, immediately on the release of GPT-3 (two years before the release of ChatGPT and the beginning of the AI boom).

And yet, as far as I can tell, _Human-like Neural Nets by Catapulting_ hasn’t yet received much public attention: one recent Hacker News thread with twelve comments, all of which are about whether human brains are anything like neural networks. Part of the reason is that (a) it’s such a long post, (b) the potted summary describes Gwern’s _claim_, but not the reasons for it, and (c) much of the beginning of the post looks like it is indeed arguing from analogy with human brains. However, I don’t think that analogy is load-bearing. Let me try and explain what I think Gwern is saying.

What is grokking?

First, let’s talk about “grokking”. In 2022, OpenAI published a paper showing that if you train a model on a simple dataset (for instance, a simple mathematical operation like division), and _keep training it_ long after the training looks like it’s stalled out, the model will suddenly make a massive jump in capability. Why does this work? The first stage of training is like rote memorization: the model has to compress as much of the training data as possible into its weights. But if you keep going, then regularization techniques (such as the pressure on the model to use smaller weight values) will motivate2 the model to find simpler and simpler ways of compressing the data. This doesn’t look like much at first (the training loss remains at zero), until the model notices that you can express the data via simply performing the underlying mathematical operation, at which point it instantly gets massively smarter. In other words, over-training a model can pressure it into actually understanding its training data. OpenAI named this process “grokking” after Robert Heinlein’s neologism, which for Heinlein means something like “gaining a deep, intuitive and fundamental understanding”3.

Gwern’s argument goes something like this:

1. Modern LLMs are worse generalizers than humans because they have not grokked their core domains 2. Grokking requires overtraining an over-parameterized model on a (relatively) small dataset, which is the exact opposite of what frontier labs do 3. However, (2) is basically how human brains learn 4. Somebody should spend a a few tens of billions of dollars3.5 on trying it, since it might immediately usher in truly human-like LLMs

I’ll skip (3), since I think the argument is still compelling without the analogy to human brains.

Are LLMs bad because they can’t grok?

I think his first point is hard to dispute. LLMs are very smart in specific areas, but they routinely make errors that humans wouldn’t make. More to the point, they routinely make errors that any human as smart as the LLM would _never_ make. This pretty clearly points to a failure of generalization: LLMs are as strong as smart humans in specific areas, but can’t generalize that intelligence to as many tasks as humans can.

Do LLMs not grok? I read through this paper that argues they do. If you graph “how much data has the LLM memorized” against benchmark performance, you can see a small initial spike in benchmark performance, followed by a big drop, followed finally by a big jump in benchmark performance. This pattern doesn’t track memorization at all: memorization increases smoothly in the background the whole time.

https://www.seangoedecke.com/static/17d0f02f0691c797be502f32fbd33a40/1d499/llm-grokking.png

I think this paper highlights the difficulty of distinguishing grokking from generalization. Obviously LLMs learn to generalize during training, and it’s plausible that learning to generalize would require a certain baseline level of memorization (so that the LLM has the raw material to generalize from). So it’s going to look like grokking.

When Gwern (and others) say that LLMs don’t grok, I think what they mean is that there’s at least one more giant generalization leap waiting to be made. Is this plausible? As an existence proof, humans are clearly capable of better generalization than LLMs. Of course, it’s _possible_ that this level of human generalization comes from features of our brain that neural networks can’t replicate, but that seems kind of ad-hoc: if neural networks can generalize at all, why would they only be able to generalize this far, and no further?

The easy examples of grokking rely on domains with a simple rule waiting to be discovered (e.g. a mathematical operation). Does human language have rules this deep? I think this is an open question, but there’s good reason to think the answer is yes. Language has deep, subtle structure: not just internal structure, but structure that reaches all the way down to the way the world is and the way human minds work.

AI labs train small-ish models on oceans of data

For the last few years, many AI researchers have been saying that data is the most important thing: that whatever model architecture you choose, with enough size and training time the model will converge to its dataset. Whether this is true or not, AI labs have spent much of their considerable resources on acquiring more, higher-quality data: from scanning physical books, paying experts to produce and label data, or partnering with companies that have a lot of data already.

AI labs have also been training _relatively_ small models. Even the largest frontier models are probably MoEs with a couple of trillion parameters and probably a tenth of that in active parameters. Of course, estimates of frontier model size are mostly guesswork, but open-source models provide a good baseline: they’re probably in the ballpark of Kimi-K3, which has just under three trillion parameters and fifty billion active parameters. That sounds like a lot, but it’s something you could probably pre-train in _a couple of days_ in the largest frontier cluster4.

Grokking requires training a huge model on a small dataset

Gwern’s prediction is that AI labs should try doing the exact opposite of what they’ve been doing. Instead of training a bunch of trillion-parameter models on massive amounts of data, try training one hundred-trillion-parameter model on a small dataset.

This sounds pretty silly on the face of it. The more data the model has access to, the smarter it will be, right? Why waste an entire training cluster on a hobbled training run? Because if Gwern is right, grokking is more likely to occur when the dataset is constrained5. If you feed the model all the data in the world, it can continue to improve simply by memorizing more new things or drawing simple connections. If the model has to ruminate on a small set of data, it’ll be forced to keep looking for deeper generalizations. You want a very large model for this so it can memorize as much of the data as possible. Every piece of memorized data can serve as raw material for generalizing.

The big labs probably haven’t done this already. Plausibly Gwern himself is enough of an insider that he would know, and so him writing this post is evidence that the labs haven’t tried it. Also, the engineering problems involved in training a hundred-trillion-parameter model have likely not been solved yet: the largest existing model is probably Claude Mythos, which is definitely not that big. But they have the resources and engineering talent to give it a pretty good shot.

Interestingly, the political obstacles might be as hard to solve as the technical ones. This training run is going to look like it failed until the moment it succeeds: training loss will drop to zero relatively quickly, then sit there for weeks or months apparently doing nothing at all to improve test loss, chewing up billions of dollars. Do any of the top players have the risk appetite or courage to keep funding this experiment all that time?

Conclusion

Gwern’s post has an extended argument that human brain development works in the same way: that human brains have far more “parameters” than frontier LLMs, and are trained on far less data6, which encourages us to make deeper generalizations in early childhood. I don’t have the background in biology or neuroscience to evaluate these claims, so I’ve expressed the case for grokking entirely without reference to it.

In 2024, it became clear to everyone that “pure scaling” — the idea that you could simply train larger and larger versions of GPT-3.5 — didn’t work. OpenAI’s “even bigger version” of GPT-4 was simply not good enough, and was eventually released as GPT-4.5 instead of GPT-5. The biggest advances since then have been reasoning, which produced another great leap forward in capability, and much better automated RL, which has ushered in the current era of reliable agents. Neither of these seem like a plausible path to artificial superintelligence.

I don’t know if I agree with Gwern or not, but forcing very large LLMs to grok is at least an idea that _could_ usher in the machine god. I can’t remember the last time I read about a simple idea this ambitious7. I hope one of the big labs tries it out.

• * *

1. For an example of the power of unhobbling, consider Claude Code or OpenClaw and the subsequent explosion of (short and long running) agentic harnesses.

↩ 2. Obviously “motivate” and “notices” are used metaphorically.

↩ 3. All of this is long before xAI’s use of the word “Grok” to name its LLMs. (Incidentally, I think this is why Gwern uses “catapulting” to describe the same thing).

↩ 4. For what it’s worth, Fable estimated the cost of Gwern’s plan at $3-10B.

↩ 5. At this model size, 25T tokens of training data at 33% utilization works out to around six million H100-hours, which a 100k GPU cluster puts out every two and a half days.

↩ 6. Two interesting pieces of contrary evidence here. First,

Obsidian intake evidence excerpt

AK-RSS-Digest（89源精选） · 2026-07-20

• status: completed
• Obsidian: /Users/gracker/Library/Mobile Documents/iCloud~md~obsidian/Documents/Obsidian/OpenClaw定时任务/AK-RSS-Digest（89源精选）/2026-07-20-AK-RSS-Digest.md
• Evidence: /Users/gracker/.hermes/evidence/ak-rss-digest/2026-07-20
• 覆盖范围: 89 个 RSS 源，最近 7 天候选 50 条；3 个 feed 抓取失败，未影响本期精选。

今日精选

1. AI Mania Is Eviscerating Global Decision-Making（8.9/10） 2. Overtraining as the path to human-like AI（8.8/10） 3. What’s the deal with all the random weekly quota resets for agents lately?（8.4/10） 4. Art Doesn’t Scale（8.3/10） 5. Plumbing Homebrew into the vulnerability ecosystem（8.2/10）

可发布正文如下：

评分：8.9/10 推荐语：这篇不是反 AI，而是在拆企业 AI 狂热怎么把决策系统改坏。最有价值的是一线咨询视角：指标被游戏化、项目不敢问效果、供应商和客户高层互相绑架，很多公司最后买到的是“贴了 AI 标签的普通项目”。 摘要：作者基于咨询和企业沟通经历，写出大组织在 AI 投资上同时缺少真实指标和纠错机制的状态：内部 chatbot 没人用、客户 chatbot 失效却不算事故、员工为了 token 指标伪装成高强度使用者。文章后半把问题落到组织政治：当客户高层也在公开宣称 100x 生产力时，供应商和内部管理者都很难讲真话。 链接：https://ludic.mataroa.blog/blog/ai-mania-is-eviscerating-global-decision-making/

• 标题：AI Mania Is Eviscerating Global Decision-Making

评分：8.8/10 推荐语：这是对 Gwern “catapulting/grokking”思路的一次清楚转述，适合跟踪下一轮大模型训练路线的人读。它把争议点说得很窄：如果当前模型缺的是更深层泛化，问题可能不在继续堆数据，而在让超大模型在较小数据集上被迫压出更简洁的规则。 摘要：文章先解释 grokking：模型在训练损失归零后继续训练，可能从记忆样本跳到掌握底层规则。作者随后讨论反向训练策略的代价：用百兆亿参数级模型、小数据集和长时间训练去赌一次能力跃迁，技术风险和组织耐心都是门槛。 链接：https://seangoedecke.com/overtraining-as-the-path-to-human-like-ai/

• 标题：Overtraining as the path to human-like AI

评分：8.4/10 推荐语：这篇把 coding agent 订阅制的一个小现象写成了产品策略观察：频繁重置额度表面上是福利，实际会改变重度用户的使用节奏和付费判断。它还指出一个竞争信号：在 Fable 5、GPT-5.6 Sol、Grok、Muse、Kimi 同时挤压市场时，额度重置也可能是防止用户试用竞品的留存手段。 摘要：作者记录了 OpenAI 在两周内多次重置 Codex 周额度，以及 Anthropic/OpenAI 在新模型发布期对额度策略的调整。文章的价值不在抱怨免费额度，而在说明 agent 产品的计费、算力调度、用户心理和竞品试用窗口已经绑在一起。 链接：https://minimaxir.com/2026/07/agent-quota-reset/

• 标题：What’s the deal with all the random weekly quota resets for agents lately?

评分：8.3/10 推荐语：这篇适合放在 AI 生成内容争论里反复引用：作者不争“AI 作品能不能让人有感觉”，而是把焦点放到创作上下文、署名、等待和稀缺性。它的判断很直接：很多人珍惜作品，不只因为结果好看，也因为有人花了时间和代价把它做出来。 摘要：作者从漫画、赝品和 AI 续作谈起，说明作品价值包含创作者经历、完成过程和读者对真实来源的信任。文章后半把“效率”问题讲清：AI 可以填满空白，但如果平台默认用廉价生成物替代人工创作，用户失去的是选择权。 链接：https://matduggan.com/art-doesnt-scale/

• 标题：Art Doesn’t Scale

评分：8.2/10 推荐语：这是一篇高质量工程复盘，讲清 brew vulns 从个人 gem 进入 Homebrew 6.0.11 的完整路径。它不只是功能发布，而是把包标识、OSV 生态注册、补丁声明、版本比较、advisory database 和 CI 边界全部摊开。 摘要：作者先说明 Homebrew 扫描 CVE 的难点：上游 repo 和版本号不够，很多 formula 带补丁，OSV/NVD 数据也经常缺少可查询的 package 信息。后文给出落地路径：新增 patch resolves 元数据、注册 pkg:brew 和 OSV Homebrew ecosystem、生成 Homebrew advisory database，并把扫描器合入 brew 主仓。 链接：https://nesbitt.io/2026/07/17/plumbing-homebrew-into-the-vulnerability-ecosystem.html

• 标题：Plumbing Homebrew into the vulnerability ecosystem

可直接发布文案

本期 AK RSS 里有几篇信号很强：一篇拆企业 AI 狂热怎么把决策机制搞坏，一篇转述 Gwern 的 grokking/overtraining 路线，一篇从 agent 额度重置看订阅产品策略，还有两篇分别谈 AI 生成内容的“来源价值”和 Homebrew 漏洞扫描的工程落地。

最推荐先读《AI Mania Is Eviscerating Global Decision-Making》：它不是泛泛吐槽 AI，而是写清了大公司里为什么越来越难诚实讨论 AI 项目的效果。接着读《Overtraining as the path to human-like AI》，换到模型训练路线层面，看下一次能力跃迁可能押在哪。

AI #Agent #LLM #软件工程 #RSS精选

备选短文案

• 企业 AI 狂热最危险的地方，不是用了多少 LLM，而是它开始奖励错误指标、惩罚诚实反馈。今天 RSS 里这篇 Ludic 文章很值得读。
• Gwern 的 catapulting/grokking 思路被 Sean Goedecke 转述得很清楚：如果模型缺的是更深层泛化，继续堆数据可能不是唯一解。
• brew vulns 这篇复盘很好看：一个小工具进入 Homebrew 主线，背后要补齐 purl、OSV、patch metadata、版本比较和 advisory database。

本期未入选说明

• Claude Code 使用 Rust 版 Bun：证据有趣，但正文偏短，适合作为技术笔记，不够本期精选强度。
• Make It Work vs. Make It Good：观点顺手，但更像短随笔，未过 8 分线。
• OpenAI 产品重组：信息有用，但原文发布时间较早，且更接近新闻报道，本期不放入精选。