Run a 2.78-trillion-parameter model on an 8GB RAM laptop
This C-based engine allows you to run a massive language model directly on a desktop computer rather than a data center. Designed to perform inference with only 8GB of memory, it handles a 1.56TB model file without requiring a GPU or complex frameworks, producing identical results using pure CPU processing.
Four strategies to overcome memory limits
The core of this project is how it handles models that do not fit in memory. First, Expert layers are read directly from disk in a 4-bit quantized state to perform multiplication, minimizing memory usage. Second, it uses KDA (Linear Attention) and MLA (Multi-head Latent Attention) to prevent memory from growing indefinitely. Additionally, the 93-layer Trunk is read in a streaming fashion so that only the necessary parts are loaded into memory, while the remaining 1.45TB of routed experts are managed with an LRU cache. These four decisions ensure identical byte-level output across various memory environments ranging from 8GB to 224GB.
Installation and execution procedure
You need approximately 1.7TB of storage space before starting, as you must store the 1.56TB checkpoint and the 109GB packed Trunk file. Linux x86-64 is the baseline, with support for macOS and Windows. The compiler must be GCC 9 or higher, or Clang 10 or higher, and OpenMP is required. You can build with make -j and verify the engine's accuracy without downloading the model by passing make test. For actual inference, use a HuggingFace token to download the model, pack the Trunk with the pack-trunk.sh script, and then run the k3 binary.
Performance and memory usage
Larger memory capacity leads to faster speeds, but the results remain the same. On a standard laptop with 8GB of RAM, it takes about 26.5 seconds per token, with the model streaming from disk. At 32GB, the time drops to 24.2 seconds, and at 64GB, it is 19.8 seconds. On workstations with 128GB or more, the model fits entirely in memory, eliminating disk wait times and speeding up to 5.6 seconds per token. The ultra preset is a minimal configuration for 8GB-class devices, maintaining precision and routing but running slower. The server preset fixes 90 of the 93 layers to provide the fastest performance.
When to choose this engine
This repository is suitable if you want to experiment with large models in environments without a GPU or where cloud costs are a burden. Written in C99, it has minimal dependencies and can be deployed lightly as a 176KB engine binary. However, it assumes you have sufficient disk space to download and store the 1.56TB model file. It is also recommended to check for CPU support of the AVX2 or NEON instruction set and the presence of a fast NVMe drive. Slow storage devices can become a bottleneck during the streaming phase, so Trunk files should be placed on the fastest local disk possible.
