Daedalus-150M: A Convolution-Attention Hybrid Designed for CPU Inference
- ID: 6820fbb6
- 原文链接: https://arxiv.org/abs/2608.20210
- PDF: https://arxiv.org/pdf/2608.20210
- 作者: Christos Koutsiaris
- 日期: 2026-08-20
- 更新: 2026-08-20
- 分类: infra
- 来源类型: arxiv
- 标签: small-model, cpu-inference, efficiency, arxiv
- 质量评分: 4/5
- 抓取时间: 2026-08-23T05:25:57Z
中文导读
Daedalus-150M 反向设计小模型：先固定目标（单用户逐 token4-bit 权重普通 CPU）再选架构，18 层中仅 6 层保留完整注意力，其余用两步卷积避免重读增长的缓存；59.9B token 从零训练后超过 GPT-2 124MPythia-160M 等同量级模型
为什么值得关注
Daedalus-150M 反向设计小模型：先固定目标（单用户逐 token4-bit 权重普通 CPU）再选架构，18 层中仅 6 层保留完整注意力，其余用两步卷积避免重读增长的缓存；59.
Grounding: CPU-first design keeps full attention in only 6 of 18 blocks, with two-timestep convolutions elsewhere; trained from scratch on 59.9B tokens; beats GPT-2 124M, Pythia-160M, OPT-125M and GPT-neo-125M.
关键信息
- 论文标题: Daedalus-150M: A Convolution-Attention Hybrid Designed for CPU Inference
- 作者: Christos Koutsiaris
- arXiv: https://arxiv.org/abs/2608.20210
- 发布时间: 2026-08-20
- arXiv 分类: cs.IR, cs.AI, cs.CL, cs.LG
- 关联标签: small-model, cpu-inference, efficiency, arxiv
English Abstract
Small language models are usually built like large ones and then squeezed onto a CPU afterwards. We did the opposite: we fixed the target first, one user, one token at a time, 4-bit weights, ordinary CPU, and chose the architecture to suit it. The result keeps full attention in only 6 of its 18 blocks. The other 12 use short convolutions whose memory is two timesteps wide no matter how long the conversation gets, so two thirds of the network never re-reads a growing cache. Trained from scratch on 59.9B tokens, the model scores 47.31 on a five-task benchmark against a bar of 42.20 that was fixed before training began. It beats GPT-2 124M, Pythia-160M, OPT-125M and GPT-neo-125M, all trained on three to six times more data, and exceeds MobileLLM-125M's published score despite that model seeing a trillion tokens. Validation bits-per-byte is 0.8685. To check the architecture rather than the training recipe, we trained a conventional all-attention model of the same size on the same data, and wrote down the winning condition before scoring either. The hybrid won the chosen quality metric by 0.81%, matched it on downstream tasks, produced a 6.3% smaller 4-bit file, and decoded 1.76x faster at 2048 tokens of context, 2.08x against an external model of similar size. In every measurement the speed advantage is near zero at an empty context and grows with length, which is what the mechanism predicts and what a merely leaner model would not show. A simple bandwidth calculation predicts only 1.17x, so memory volume alone does not explain the gap. We also report what did not work: an unmitigated 4-bit quality cost, roughly half the convolution channels ending up inert and impossible to remove, and a vocabulary larger than this model size warrants.
English Summary
Daedalus-150M designs a small LM CPU-first: the target is fixed in advance (one user, one token at a time, 4-bit weights, ordinary CPU), full attention is kept in only 6 of 18 blocks, and the rest use short two-timestep convolutions that never re-read a growing cache. Trained from scratch on 59.9B tokens, it beats GPT-2 124M, Pythia-160M, OPT-125M and GPT-neo-125M.
Obsidian Notes
- 内容由
opencli arxiv paper拉取 arXiv 元数据与摘要生成。 - 中文导读与价值判断均锚定在条目已有摘要、论文摘要、作者、日期与分类信息上；未补充论文摘要之外的实验细节。