One argsort, up to 17% faster: the free MoE inference speedup

The cheapest performance win I’ve seen all month comes from a Hugging Face community post. If you run offline batch inference with mixture-of-experts models, sort your prompts before generating. Grouping similar requests so they decode together made stock vLLM 12 to 17% faster on Qwen3-30B-A3B and 7 to 12% faster on gpt-oss-120b in the author’s tests.

No model change. No kernel change. No custom scheduler. One argsort call before generate.

If you run bulk evals, synthetic data generation, or labeling pipelines, that is free throughput. The mechanism underneath is worth understanding too, because it explains a lot about why MoE inference costs behave so erratically.

MoE decoding is mostly a memory problem

During mixture-of-experts decoding, the GPU spends most of its time reading weights, not computing. Every expert that any request in the batch routes to has to stream from HBM once per layer. The arithmetic is cheap. The memory traffic is not.

Routing decisions from different requests barely overlap. The author measured a batch of 32 random requests touching about 94 of Qwen3-30B-A3B’s 128 experts. You pay to stream nearly the whole model every step, because one request somewhere in the batch wanted each expert. A model with 30B parameters where each token only uses a fraction of them suddenly doesn’t feel so sparse.

This is also why your cost per request swings depending on what else is in the batch. Two identical prompts run alone are cheap. The same two prompts mixed into a grab-bag batch with code, French, and news articles drag every expert those other domains need into memory alongside them.

Expert choice isn’t random

Routing looks chaotic at the batch level, but per domain it’s tight. Code tokens concentrate their routing on one cluster of experts. French tokens prefer a different cluster. Measured across domains, each one locks 68 to 80% of its routing decisions onto its favorite 32 experts out of 128.

So a mixed batch streams experts for every domain present. Sort the batch so code requests decode together, and the per-step expert count drops 19 to 25%. Decode time follows almost linearly: roughly 2.9 microseconds per additional expert touched.

Same model, same hardware, same answers. The batch just stops paying for experts nobody at that step needed.

I find that result oddly satisfying. The structure was sitting there in the routing tables the entire time. Nobody looked because expert routing is supposed to be an opaque learned artifact, and opaque learned artifacts are not where you expect to find a sortable key.

What to do now

This applies to one specific workload: offline batch jobs where all prompts are known up front. Eval runs, synthetic data generation, classification over a corpus. If that’s you, the recipe is short.

1. Confirm the job is offline batch. Latency-sensitive serving doesn’t benefit, since you can’t hold a live request until its neighbors arrive. This trick trades scheduling freedom for throughput, and only batch jobs have that freedom to trade.
2. Sort by domain if you have labels. All-code corpus, single-language support tickets, one product’s reviews. Grouped requests showed the full 12 to 17% on Qwen3-30B-A3B in stock vLLM.
3. Cluster first if you don’t have labels. Mixed chat traffic can be grouped with a small embedding model. The author used a 0.6B embedder, which still returned +5% on Qwen3 and up to +4% on gpt-oss-120b. You spend a little on embedding to save a lot on decoding.
4. Measure on your own traffic. Routing concentration varies with domain mix. Expect to land somewhere between the clustering floor and the labeled-sort ceiling.

The numbers were measured end to end in vLLM 0.27 on a single AMD MI300X, not in a microbenchmark. The speedup survives contact with the real serving stack.

What 15% actually buys you

Throughput percentages can feel abstract, so make it concrete. A 15% speedup on an 8-GPU batch job returns roughly 1.2 GPUs’ worth of capacity. On a cluster you rent by the hour, that’s one machine in eight you stop paying for, or one in eight of extra eval capacity you get before the next model launch deadline. Synthetic data pipelines that run nightly suddenly finish 90 minutes earlier on a 10-hour run.

None of that requires believing the peak numbers. Even the floor case, +4% from embedding-based clustering on gpt-oss-120b, is real money at inference scale.

Why this post is worth reading beyond the speedup

Two reasons.

First, it’s the clearest explainable tour of MoE routing I’ve come across. If you’ve wondered why mixture-of-experts inference costs jump around so much between batch compositions, “distinct experts touched per step” is the mental model that makes it click.

Second, it’s a profile-first engineering story, and those are rare. The author didn’t set out to find a sorting trick. They profiled decoding, found expert streaming dominating the timeline, then went looking for structure in what looked like noise. The optimization fell out of understanding the system rather than out of tuning knobs. That habit transfers to whatever stack you run.

The original post has the full methodology and plots, and it’s worth reading directly. The next time an inference bill looks like a fixed cost, remember that sometimes the fix is embarrassingly close to a sorted list.