Run AI agent inference with minimal latency
When AI agents require immediate responses, you can operate an inference engine designed to minimize latency. TokenSpeed is an open-source project that separates control logic from execution logic to balance stability and development speed. It supports NVIDIA and AMD hardware and has officially joined the PyTorch ecosystem.
Separating control and execution
Unlike existing engines, TokenSpeed clearly distinguishes between the control plane and the execution plane. Core logic such as request lifecycle and memory management is handled by a finite state machine written in C++, with type systems at compile time ensuring the safety of resource reuse. In contrast, actual model execution is done in Python, allowing researchers and engineers to modify and test code quickly. This design aims to maintain the accuracy of core scheduling while improving development efficiency at higher levels.
A variable kernel system
Kernels are treated as independent modules rather than part of the engine. Through a public API and a centralized registry, you can swap kernels for various accelerators like plugins. It specifically includes an MLA (Multi-Head Latent Attention) implementation optimized for agent workloads on the Blackwell architecture, enabling high performance on the latest hardware.
Recent updates and performance metrics
As of October 2026, it recorded a performance of 685 tokens per second per user when serving the Qwen3.8-Flash-Next model on NVIDIA GB300. It is also expanding hardware versatility by supporting Kimi K3 inference on AMD Instinct MI355X GPUs.
Neutral governance
TokenSpeed is sponsored by the non-profit LightSeek Foundation and maintains a neutral position independent of specific commercial companies. It is sometimes described as the 'Switzerland of open-source LLM inference engines.' While major hardware vendors like NVIDIA and AMD provide engineering support, the foundation determines the project's direction.
Considerations before adoption
This engine shows particular strength in agent workloads, environments where short responses are exchanged repeatedly. It is suitable if you want performance at the level of TensorRT-LLM and usability at the level of vLLM simultaneously. However, because it is optimized for specific hardware (Blackwell, MI355X, etc.) and models (Qwen, Kimi, GLM, etc.), you should first check if your hardware and model combination is included in the 'Model Recipes' in the official documentation. Compatibility with the PyTorch ecosystem is guaranteed, but migrating directly from existing vLLM or TensorRT-LLM environments may require adjustments due to parameter differences.