SpecBox: Speculative Sandbox Scheduling for Efficient LLM Agent Serving

Source: https://arxiv.org/abs/2607.23933

Authors: Yihui Zhang, Tianyu Wo, Jinghao Wang, Xiaoyang Sun, Menghao Zhang, Cangzhou Yuan, Li Li, Chunming Hu, Albert Y. Zomaya, Renyu Yang

Published: 2026-07-27

Categories: cs.DC, cs.AI, cs.LG, cs.PF

Abstract

As LLM agents increasingly rely on the Model Context Protocol (MCP) to invoke isolated external sandboxes, disaggregated sandbox deployment introduces a fundamental tension between resource utilization and interactive tail latency. Persistent long-lived sandbox reservations incur excessive memory overhead at scale, while lazy on-demand instantiation generates severe cold-start penalties that degrade response performance under multi-tenant, multi-turn agent workloads.

SpecBox is a runtime built around speculative sandbox preallocation tailored for dynamic LLM agent execution pipelines. It implements keyword matching and streaming semantic embedding to enable intent-driven sandbox prewarming, which identifies pending tool execution demands mid-LLM token generation and fully overlaps sandbox bootstrapping with model inference. To extend prewarming windows across sequential agent steps, the framework leverages context-aware stochastic prefetching atop a sandbox dependency graph.

SpecBox cuts P99 end-to-end latency by up to 2.9× relative to the on-demand sandbox baseline, while slashing peak memory consumption by 45.9% compared to permanently reserved sandbox deployments.

Evidence (Obsidian)

From 论文流水线 2026-07-31:

MCP sandbox 投机预热，把 bootstrap 和推理重叠，治 cold-start。
定位是 serving 侧 runtime，直接面向 MCP 工具调用密集的 Agent 负载。