smevals: a small eval suite for models, prompts, and harnesses
Source: https://simonwillison.net/2026/Jul/31/smevals/
Author: Simon Willison
Date: 2026-07-31
Summary (Chinese)
Simon Willison 与 Prime Radiant 合作开发的 eval 框架。核心是一套清晰的词汇表：eval 是挑战集合，task 是单个挑战，config 指定模型+参数+harness，run 记录执行结果，grader 用 check 序列打分。运行和评分分离设计，可以先批量跑再统一评。支持本地 web 可视化和静态 HTML 报告导出。用 uvx smevals docs 就能让 coding agent 自学使用方式。
Summary (English)
An eval framework developed with Prime Radiant. Core vocabulary: eval (challenge collection), task (single challenge), config (model+params+harness), run (execution result), grader (check sequences). Run and grade operations are separated. Supports local web visualization and static HTML export. Self-documenting via uvx smevals docs.
One-liner
运行和评分分离的轻量 eval 框架，第三次迭代设计
*Imported from Obsidian digest notes on 2026-08-01. Content grounded in fetched source metadata via opencli.*