Research papers, made simple to understand

• Serving & Infrastructure

TokenRouter boosts token-level LLM routing throughput by up to 64.15× with async execution

TokenRouter is a serving system for token-level routing in large language models that uses asynchronous model-centric subservers and delayed batching, achieving 2.01–64.15× higher decoding throughput and lower latency than prior systems across diverse algorithms and workloads.

Paper: TokenRouter: Efficient Serving System for Token-Level LLM Routing · Tianyu Fu, Tengxuan Liu, Ruoxi Wang and 4 others

106 upvotes on Hugging Face9 Oct 20266 min readRead the article