Every AI gateway pitch sounds roughly the same: why pay frontier prices for every prompt when our smart routing sends each one to the best model for the job? It’s a compelling pitch. Volume inference is expensive, the model menu keeps growing, and picking per-prompt sounds like obvious savings.
An audit published this week by AutoBench tested that promise and came back with an uncomfortable answer. Out of seven commercial and open-source LLM routers, across seventeen headline configurations, not one beat the most boring strategy imaginable: always calling a single mid-priced model.
Not “most fell short.” All of them.
What AutoBench tested
The methodology deserves a minute of attention, because router benchmarks are easy to game. AutoBench pre-registered its evaluation before running it, scored a pool of twenty models on its own quality metric, then measured each router on two axes at once: answer quality and cost per question. The full data and leaderboard are public at autobench.org, and Peter Kruger’s writeup is on the Hugging Face blog.
That pre-registration matters. It means the test questions and scoring rules were locked in before anyone knew which routers would look bad.
The prize nobody captured
To size the opportunity, AutoBench computed what a router with perfect knowledge would achieve: 100% Delta-Hit at $0.0018 per question. Always calling GPT-5.6 Sol, a strong mid-priced option, gets 87.5% at $0.079 per question. That is roughly a 44x cost gap for 12.5 points of quality, and it’s the theoretical prize for routing well. It is enormous.
The good quadrant (better than random quality, cheaper than a random pick) contained zero commercial products. Every router tested either gave up quality, spent more, or both.
The knob in front of a constant
The most damning detail is structural. Five of the seventeen router configurations are effectively constants: a single model receives at least 90% of the traffic. The “quality knob” in those settings doesn’t gradually tune anything. It switches which constant you get.
So the picture many buyers carry, a clever dispatcher weighing each prompt and sending it to the right brain, doesn’t match what’s running in production. In several cases it’s a fixed model choice with a settings panel bolted on the front.
Routers don’t even agree with each other
If routing were a solved science, independent routers given the same prompt would mostly agree. They don’t. Across 59 cross-vendor pairs, the median overlap in model choice was zero. Median. Not “low.” Zero.
Two routers can read the identical prompt and each be confident a different model is the right call. That suggests dispatch decisions have more to do with each vendor’s internal defaults than with any ground truth about the prompt itself.
Two caveats to take seriously
Fairness first: AutoBench scored quality with its own methodology over its own 20-model pool, and router vendors will dispute that scoring. A router tuned against a different benchmark can look better. Pre-registration reduces the gamesmanship but it’s still one lab’s yardstick.
Second, the boring strategy isn’t optimal either. Always calling your workhorse model still loses to the perfect oracle by a wide margin. The finding isn’t “routing is impossible.” It’s that nothing currently on the market gets close to what routing promises.
What this means for content and marketing ops
If you run volume generation, SEO drafts, product copy, transcript processing, classification at scale, you are exactly who routers are selling to. The pitch lands because your bills are real. But this audit says the premium you pay for intelligent dispatch probably isn’t buying much intelligence.
The open-weight tier strengthens the case for boring. Mistral’s Large 4, released on Hugging Face this week, is the latest in a line (Kimi, GLM, Qwen) that lets you pin one strong model to fixed infrastructure and stop paying per-token premiums for steady workloads. A constant you chose on purpose beats a constant pretending to be a decision.
What to do this week
- Pull a week of your gateway logs and compute what fraction of prompts went anywhere other than the top model. If one model takes 90% or more, you are running a constant.
- Replay a sample of real prompts through your workhorse model directly, and compare quality and cost against the router’s picks on the same inputs.
- If the difference is noise, and per this audit it likely is, collapse to the single model, renegotiate the contract, or self-host.
- Before renewing any gateway deal, ask the vendor one question: what fraction of our traffic did you actually reroute last month, and what did it save us versus one fixed model? Make them show numbers, not a roadmap.
Routing is a genuinely hard problem with a genuinely large prize, and a market grew up around the prize before anyone solved the problem. Until the leaderboard changes, the smartest routing decision available is checking whether you need a router at all.
