Optimization

One argsort, up to 17% faster: the free MoE inference speedup

Sorting prompts by domain before batch generation made stock vLLM up to 17% faster on MoE models, with no model or kernel changes. Here is why it works and how to apply it.