Friday, 9 October 2026 · 6 min read
TokenRouter boosts token-level LLM routing throughput by up to 64.15× with async execution
TokenRouter is a serving system for token-level routing in large language models that uses asynchronous model-centric subservers and delayed batching, achieving 2.01–64.15× higher decoding throughput and lower latency than prior systems across diverse algorithms and workloads.
Based on the paper TokenRouter: Efficient Serving System for Token-Level LLM Routing by Tianyu Fu, Tengxuan Liu, Ruoxi Wang, Yixin Dong and 3 others
Challenges of efficient token-level routing in LLM serving
Large language model (LLM) routing distributes inference across multiple models to improve cost and quality trade-offs. While coarse-grained routing at the session or query level is common in production, recent research shows that routing at the token level—choosing which model generates each token—can yield better efficiency and quality.
Token-level routing exploits the fact that different tokens in a single request vary in difficulty, allowing smaller models to generate easy tokens and larger models to handle hard tokens, or enabling models with complementary expertise to collaborate within one response.
However, existing serving systems assume all requests run synchronously on a single model, causing three main problems for token-level routing: step desynchronization (models have different per-step latencies, causing idle time), batch admission delay (requests wait to be admitted into batches when switching models), and high implementation complexity due to lack of interfaces for per-token routing.
Sign in to read the whole paper
You have read the opening. The rest is free with a Bezen account, and no card is needed.
In the rest of this paper
- How the method works, step by step, with a flow chart
- The key formula from the paper, explained symbol by symbol
- Two figures from the paper, and how to read them
- How it was tested, what was found, and where it falls short