Everyone is building LLM routers, we deprecated ours
- ID: b5a2b7d3
- Source: Manifest Blog
- Original URL: https://manifest.build/blog/why-we-deprecated-our-llm-router/
- Author: Bruno Perez
- Published: 2026-07-31
- AAIF category: infra
- Tags: llm-routing, llm-gateway, agent-infra, prompt-caching, model-selection, observability
- Quality score: 4/5
- Fetched at: 2026-08-03T23:34:11+08:00
- Obsidian evidence: OpenClaw定时任务/内容选题编排/2026-08-03-内容选题编排.md
One-liner
Manifest 的 LLM router 复盘提醒 Agent 基础设施降本不能只看单次调用价格。
Chinese summary
Manifest 复盘自己下线 LLM router 的原因：按 prompt 预判复杂度无法覆盖工具调用后的真实任务上下文；prefix cache 对系统提示和历史上下文的降本效果往往比路由更直接；多模型切换会破坏行为一致性，并让 eval、提示词和可观测性维护成本上升。文章基于四个月、约 7000 个云端用户的使用经验，主张多数场景应显式选择稳定模型和参数，而不是把不确定性外包给 router。
English summary
Manifest explains why it deprecated its LLM router after four months across roughly 7,000 cloud users. The post argues that prompt-only routing misses complexity discovered through tool calls and repo context, prefix caching can reduce cost more reliably than switching models, and dynamic routing hurts behavior consistency while adding evaluation, prompt, and observability overhead.
Grounded source excerpt
Everyone is building LLM routers, we deprecated ours
作者: Bruno Perez
发布时间: 2026-07-31T00:00:00Z
原文链接: https://manifest.build/blog/why-we-deprecated-our-llm-router/
https://twitter.com/intent/tweet?url=https%3A%2F%2Fmanifest.build%2Fblog%2Fwhy-we-deprecated-our-llm-router%2F&text=Everyone%20is%20building%20LLM%20routers%2C%20we%20deprecated%20ourshttps://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fmanifest.build%2Fblog%2Fwhy-we-deprecated-our-llm-router%2F
We don’t believe in model routing anymore. For most use cases, sticking to a single battle-tested model is the best thing you can do.
Recently, there’s been huge hype around AI model routers that select the model that will respond to your request on the fly. There have been many launches in recent weeks with similar promises of reducing inference costs. We had our LLM router too, and decided to remove it.
Some context first: we launched the Manifest LLM router in March as a key feature in our LLM gateway, and we deprecated it in June, shutting it down for good on September 1st. Our router was classifying each request into one of four different tiers of complexity: simple, standard, complex and reasoning.
Like most LLM routers, ours was made for cost reduction. Why call a powerful, and therefore expensive, model for a simple task? Routing to the most cost-effective model seems like a natural solution, right? Not that simple. After four months of usage across 7000 cloud users, we saw mixed results and a lot of GitHub issues and discussions about it. Let’s dive into the main problems.
Complexity cannot be deduced from the prompt alone
The prompt alone does not contain the whole task; it is just the trigger. A lot of the context that determines complexity is only discovered later through tool calls, web searches, and so on.
Let’s take an example: _“evaluate the tests for the repo $GIT\_REPO and improve them”_ can be a very simple task if you mention a personal website written in plain HTML5; or an incredibly complex task if you target the Linux kernel repo.
Cache is more effective than routing for reducing costs
Cache reads are between 75% and 90% cheaper than uncached inputs. System prompts and conversation history often represent a lot of tokens. Prefix cache works extremely well for those because they sit at the beginning of the prompt.
A cache-aware model router will take that into account by adding stickiness to the initially chosen model and keeps querying it. In other words, the router will do its job by, ironically, _not doing it_.
LLM routers break behavior consistency
Some say that “engineers should not be concerned about choosing the best LLM for their task”. Well, we strongly disagree.
Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models. At Manifest, every engineer selects models and effort parameters based on their intent.
Jumping from a model to another during working sessions results in lower quality of the overall work, and detaches the people from mastering their tools.
Unpredictability has a cost
No one likes unpredictability, especially software engineers.
In automated agentic workflows or autonomous agents, managing that extra layer of uncertainty can cost more than it saves. Think of evals, system prompts, observability and so on. Everything suddenly becomes harder to maintain.
Isolating different requests and setting up the right models, params and prompts for it seems naturally superior in most cases.
Conclusion
There are probably many use cases where LLM routing can be useful, and the companies that launched those have probably good reasons.
However, based on our experience, we’ve concluded that in most of the use cases we saw, it was not worth it. The amount saved is paid somewhere else, and that cost is harder to estimate.