WCXB: A Multi-Type Web Content Extraction Benchmark
Source: https://arxiv.org/abs/2605.21097 Authors: Murrough Foley Published: 2026-05-20 Updated: 2026-05-20 Categories: cs.CL
中文摘要
WCXB 构建了一个多类型 Web 内容抽取基准, 覆盖 2,008 个页面, 1,613 个域名和七种结构化页面类型. 摘要指出当前抽取器在文章页上表现接近, 但在产品, 列表, 文档等结构化页面上 F1 差异明显. 这对 RAG, 搜索索引和训练数据清洗都有实用参考.
英文摘要（Abstract）
Web content extraction - isolating a page's main content from surrounding boilerplate - is a prerequisite for search indexing, retrieval-augmented generation, NLP dataset construction, and large language model training. Progress in this area has been constrained by the limitations of existing evaluation benchmarks, which are small (100-800 pages), restricted to news articles, or based on web pages from over a decade ago. We introduce the Web Content Extraction Benchmark (WCXB), a dataset of 2,008 web pages from 1,613 domains spanning seven structurally distinct page types: articles, forums, products, collections, listings, documentation, and service pages. The dataset includes a 1,497-page development set and a 511-page held-out test set with matched page type distributions. Ground truth annotations were produced through a five-stage pipeline: LLM-assisted drafting, automated verification, four-pass frontier model review, snippet and quality verification scripts, and human review. We evaluate 13 extraction systems - 11 heuristic and 2 neural - and find that while top systems converge on articles (F1 = 0.93), performance diverges sharply on structured page types (F1 = 0.41-0.84), revealing blind spots invisible to existing article-only benchmarks. The dataset is released under CC-BY-4.0 with HTML source files, ground truth annotations, page type labels, and baseline results.
一句评点
WCXB 用多类型真实网页暴露了文章专用抽取基准的盲区.
来源与元数据
- arXiv: https://arxiv.org/abs/2605.21097
- PDF: https://arxiv.org/pdf/2605.21097v1
- Authors: Murrough Foley
- Published: 2026-05-20
- Updated: 2026-05-20
- Categories: cs.CL
- Comments: Dataset: github.com/Murrough-Foley/web-content-extraction-benchmark, doi.org/10.5281/zenodo.19316874. Leaderboard: webcontentextraction.org. Preprint also deposited at doi.org/10.5281/zenodo.19664685
本文件由 content-fetcher 通过 opencli arxiv paper 获取论文元数据和摘要后回填，不包含未读取全文的额外推断。