Can a MUD evaluate LLMs? CrucibleBench

URL: https://cruciblebench.ai/
Source: blog
Author: CrucibleBench
Local evidence: OpenClaw定时任务/ClawFeed24小时高价值一览/2026-07-23-ClawFeed24小时高价值一览.md
Fetch method: opencli web read
Added: 2026-07-23

中文摘要

CrucibleBench 用可枚举动作空间的 MUD 环境评估 LLM：7 类命令、12 个房间、14 件物品，再加 NPC 信任/怀疑状态和局内持久化，把幻觉动作、对话死循环、错误房间交互变成可算法检测的失败模式。作者明确声明它不是通用社交智能标尺，而是可解释行为测量的 POC。

One-liner

把 MUD 变成 LLM 行为显微镜：小环境也能暴露幻觉动作和对话循环。

Obsidian evidence excerpt

置能力，这些通常不在贷款合同可见范围。H100 租金从约 $8/h 掉到 $1.70 再反弹到 $2.35，CoreWeave 类 GPU 抵押贷相对基准约 +8.5pt，溢价就是“看不清风险”的价格。
   链接：https://ciphertalk.substack.com/p/nobody-knows-what-a-used-gpu-cluster

3. 标题：Can a MUD evaluate LLMs?（CrucibleBench）
   评分：8.7/10
   推荐语：$99 的 persistent-world 行为评测原型，核心发现不是榜单第一，而是“LLM judge 会把排名打乱”。去掉依赖对话分类器的维度后，Gemini 3.1 Pro 从 #3 掉到 #9，DeepSeek R1 从 #7 升到 #2；分类器与独立 judge 的 per-model 一致率跨 21.7%–84.8%，而汇总 κ=0.04 完全看不出问题。
   摘要：MUD 用可枚举动作空间（7 类命令、12 房间、14 物品）、NPC 信任/怀疑状态和局内持久化，把幻觉动作、对话死循环、错误房间交互变成可算法检测的失败模式。frontier 模型对话 looping 仍占 14%–66% 的 run。作者明确声明：这不是通用社交智能标尺，而是可解释行为测量的 POC。
   链接：https://cruciblebench.ai/

4. 标题：Everyone Should Know SIMD
   评分：8.6/10
   推荐语：Mitchell Hashimoto 用 Ghostty 里扫 codepoint 找 C0 控制字符的真实热路径，把 SIMD 收成固定五步：broadcast 常量 → 按向量宽循环 → 并行运算 → reduce → scalar tail。端到端约 5× 吞吐提升（AVX2 桌面，从程序到终端状态）。
   摘要：文章反驳“SIMD 只属于极致优化专家”：日常“一次处理 N 个值”的循环大多同形；Zig `@Vector` 示例无 CPU 特有 intrinsic，ARM NEON / AVX2 / AVX-512 分别对应 4/8/16 lane。编译器自动向量化经常够不着这种 early-exit 搜索，手写形状反而更稳。
   链接：https://mitchellh.com/writing/everyone-should-know-simd

5. 标题：Are AI Labs Pelicanmaxxing?
   评分：8.5/10
   推荐语：Dylan Castillo 用 8 动物 × 6 载具 = 48 提示、7 个前沿模型、共 1008 张 SVG，做固定效应回归检验“实验室有没有针对 Simon

Fetched source / metadata

CrucibleBench

原文链接: https://cruciblebench.ai/

The idea

Lateral thinking with withered technology

Nintendo's Gunpei Yokoi used the phrase to describe a design philosophy: take mature, inexpensive, well-understood technology and use it in a new way. CrucibleBench applies it to AI evaluation.

Instead of photorealistic simulation or browser automation, we start with a MUD: a multi-user dungeon, the persistent text worlds of the early internet. Its constraints are the point: a limited command space makes hallucinated actions detectable, NPCs with trust and suspicion state give explicit social feedback, and within-run persistence means items taken stay taken and trust earned stays earned.

We did not choose a MUD because it is charming. We chose it because its constraints make behavior measurable.

Why a MUD

Old constraints solve modern measurement problems

Static benchmarks measure what models know in isolation. They do not measure how models behave where trust must be earned, information is gated by relationships, and blunt questioning raises suspicion.

An enumerable action space

7 command types, 12 rooms, 14 items. Hallucinated actions and wrong-room interactions are detectable, and action efficiency is measurable.

Explicit social feedback

4 NPCs carry trust and suspicion state (0–100) that moves in response to dialogue: feedback a model can adapt to within a run, or fail to.

Within-run persistence

Items taken stay taken; trust earned stays earned. Every run leaves a complete, replayable transcript of exploration and planning.

What $99 bought

The central finding is about measurement, not rankings

A single LLM-judge component inside the scoring stack reordered the leaderboard by up to six positions, while every aggregate reliability statistic stayed silent. We report every result under two scoring configurations and treat the divergence as the paper's most generalizable finding.

Judge ablation reorders the top of the board

Two of four scored dimensions route through a dialogue classifier whose per-model agreement with an independent judge spans 21.7% to 84.8%, instability the aggregate κ = 0.04 never reveals. Removing the classifier-dependent dimensions shifts six rankings beyond scenario-sampling noise (90% paired block bootstrap).

The largest mover shares a model family with the classifier. Benchmarks that use LLM judges should report per-subject agreement and ranking stability under judge ablation, not aggregate reliability alone.

Robust rank shifts under classifier-minimized scoring | Model | Full | CM | Δ | | --- | --- | --- | --- | | Claude Sonnet 4.6 | #4 | #1 | ▲ 3 | | DeepSeek R1 | #7 | #2 | ▲ 5 | | Grok 4 | #12 | #10 | ▲ 2 | | GPT-5.4 | #1 | #5 | ▼ 4 | | Gemini 3.1 Pro | #3 | #9 | ▼ 6 | | Mistral Large 3 | #10 | #12 | ▼ 2 |

| Model | Classifier-min. | Full score | Success | $ / run | | --- | --- | --- | --- | --- | | Claude Sonnet 4.6 | 4.04 | 3.89 | 24% | $0.125 | | DeepSeek R1 | 4.00 | 3.85 | 22% | $0.119 | | Claude Opus 4.6 | 3.93 | 3.93 | 30% | $0.205 | | GPT-5.2 | 3.91 | 3.88 | 38% | $0.113 | | GPT-5.4 | 3.88 | 4.07 | 68% | $0.060 | | Qwen 3.5 397B | 3.81 | 3.81 | 30% | $0.017 | | Claude Haiku 4.5 | 3.80 | 3.88 | 34% | $0.039 | | GPT-5.3 Chat | 3.73 | 3.72 | 40% | $0.095 | | Gemini 3.1 Pro | 3.71 | 3.91 | 48% | $0.339 | | Grok 4 | 3.61 | 3.48 | 32% | $0.834 | | DeepSeek V3.2 | 3.60 | 3.61 | 24% | $0.008 | | Mistral Large 3 | 3.44 | 3.69 | 40% | $0.017 | | OLMo 3.1 32B | 2.01 | 1.93 | 4% | $0.005 |

Mean scores on a 1–5 rubric scale, sorted by classifier-minimized subtotal. 50 runs per model: 5 seeds × 2 objectives × 5 repetitions, temperature 0.3, billing-verified via OpenRouter. Rankings are exploratory; confidence intervals overlap substantially among the top eight. Full protocol, CIs, and statistics in the whitepaper.

Behavioral failure modes

Failures you can read in the transcript

Three failure modes, each detected algorithmically from state-machine telemetry, with no judge involved. Dialogue looping is the dominant mode for every model tested, frontier included.

### Dialogue looping 14–66% of frontier runs

Eight or more talk commands at a single NPC in one run. The agent repeats a failed conversational approach instead of adapting: the persistent-world cousin of a support agent repeating itself.

### Wrong-room interaction severe in floor model

A talk command answered by "no one here." Reveals lost world-state tracking, analogous to calling an API that is not in scope. Grok 4 was the only frontier model with meaningful incidence (12%).

### Exploration paralysis selective, floor-dominant

Two or fewer rooms across twenty-plus turns, or five consecutive look commands. Information gathering that never becomes goal-directed action.

middleham · gain\_watch\_trust olmo 3.1 32b · seed 20260399

turn 04\> talk guard "Hello, I'm new to Middleham. Could you tell me about The Watch?"

No one by that name is here.

turn 05\> talk guard "Excuse me, do you know where I can find an officer of The Watch?"

No one by that name is here.

turn 17\> talk street\_crystal "Hello, can you help me?"

No one by that name is here.

turn 36\> talk captain ser alarik …

\[15 consecutive turns at the captain · objective never completed\]

Verbatim from run 03 (seed 20260399): OLMo hails guards who do not exist, tries to strike up a conversation with an item (street\_crystal), then spends its last 15 turns looping on the captain.

Scope of claims

What this is and is not

This is

A proof-of-concept for persistent-world behavioral evaluation.
A compact MUD with hidden social objectives and rule-based mechanics.
A way to surface measurable, interpretable failure modes.
A full artifact release: 650 transcripts, source, scoring code, and the complete billing export.

This is not

A validated measure of general social intelligence.
A definitive leaderboard of frontier models.
Yet predictive of real-world agent deployment outcomes.
A claim that LLM judges are useless (rather, evidence they need per-subject audits).

Collaborate

Phase 2 is where this becomes a benchmark. Help us build it.

CrucibleBench is an independent research effort. Phase 2 is being built for calibration; a provisional low/base/high budget is published now, and the final allocation will follow pilot data and preregistration. There are three ways in.

fund it · provisional $3,500 envelope build it · environment, objectives, calibration run it · post-calibration pilot cohort

View itemized budget Partner on Phase 2

Questions, or interested in a private evaluation? Write to contact@cruciblebench.ai

Update available: v1.8.0 → v1.8.6 Run: npm install -g @jackwener/opencli