Run 290B-class frontier models at conversational speed on gaming PCs
Experience data-center-level intelligence on hardware you already own. Imagine running massive open-weight models locally for immediate responses. FreeToken is an edge-native MoE serving engine designed to execute frontier-scale models on personal hardware.
Execution that unifies hardware resources
The engine treats the GPU, CPU, host memory, and interconnect as a single elastic inference platform. It applies a bandwidth-adaptive CPU-GPU co-execution policy and full-layer double-buffered prefill streaming. This structure maintains high throughput even on consumer hardware.
Context management without redundant computation
Semantic caching sets semantic anchor checkpoints in repetitive states and the KV cache. Agentic context editing avoids unnecessary context recalculation when tool calls or thought blocks occur. VRAM between the expert cache and KV memory can be dynamically reallocated at runtime. Memory is adjusted flexibly without restarting the engine or reloading weights.
Supported models and hardware scope
It supports frontier open-weight MoE models such as DeepSeek-V4-Flash, Qwen3.6-35B-A3B, and GLM-5.2. It handles various quantization formats including MXFP4, NVFP4, FP8, and BF16. Anthropic and OpenAI compatible APIs allow easy integration with real coding and tool-calling agents like Codex, Claude Code, OpenCode, OpenClaw, and DeepSeek Harness. It natively supports NVIDIA RTX 30, 40, and 50 series GPUs, extending to consumer laptops, gaming desktops, and workstation GPUs.
Installation and startup
Download the desktop app for Windows or Linux from flashml.ai to automatically prepare engine settings and the GUI. You can run models, chat, and tune the engine via the graphical interface. If you prefer the CLI, install it using uv or pip. Run the command uv pip install "freetoken[accel]". To build from source, clone the repository, set up a virtual environment, and run uv pip install -e ".[accel]".
Pre-use considerations
This tool natively supports NVIDIA RTX 30, 40, and 50 series GPUs. Compatibility with other GPU architectures is not specified in the README. It follows the Apache License 2.0, and citing the paper is required for research use. It acknowledges design and code references from mini-sglang, SGLang, vLLM, FlashInfer, flash-linear-attention, LightLLM, and llama.cpp. Exact performance figures for specific models may vary by hardware configuration, so testing in your actual environment is recommended.
