TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving
Source: https://arxiv.org/abs/2607.29678
Authors: Zhenyu Zhang, Zhichao Cao
Published: 2026-07-31
Categories: cs.CL, cs.DC, cs.PF
Grounding: opencli arxiv metadata + Obsidian paper digest excerpt (OpenClaw定时任务/论文流水线/2026-08-04-论文流水线.md)
摘要（中文）
OpenCLI arXiv abstract: LLM serving stacks cache prompt KV state, yet the front end still re-tokenizes the full request text on every call. Coding agents pay the most: each call resubmits a long transcript after a small append, and reuse is hard because a short append can move token boundaries near the end of the prior sequence. Across 153,951 agent calls, the median append is 1.4K characters; only 1.0-3.6% of calls start or rebuild a session, but those carry multi-million-character contexts.
Abstract
LLM serving stacks cache prompt KV state, yet the front end still re-tokenizes the full request text on every call. Coding agents pay the most: each call resubmits a long transcript after a small append, and reuse is hard because a short append can move token boundaries near the end of the prior sequence. Across 153,951 agent calls, the median append is 1.4K characters; only 1.0-3.6% of calls start or rebuild a session, but those carry multi-million-character contexts. At the fleet's 94.1% prompt-cache hit rate approaching 0.99, tokenization grows from 10% to 64% of time to first token. TokTier is a stateful CPU+GPU tokenization service for this two-mode workload with one contract: emitted token IDs are always identical to full reference tokenization of the request text.
Intake rationale
- Category: infra
- Quality score: 5/5
- Why it matters: TokTier shows tokenization can dominate TTFT once agent prompt-cache hit rates get high.
Obsidian evidence excerpt
OpenCLI arXiv abstract: LLM serving stacks cache prompt KV state, yet the front end still re-tokenizes the full request text on every call. Coding agents pay the most: each call resubmits a long transcript after a small append, and reuse is hard because a short append can move token boundaries near the end of the prior sequence. Across 153,951 agent calls, the median append is 1.4K characters; only 1.0-3.6% of calls start or rebuild a session, but those carry multi-million-character contexts.