Run Tens of Billions of Parameters on Your Local Machine Without a Data Center
This tool runs large AI models directly on your desktop by keeping only parts of the model in memory and reading the rest from disk as needed. It automatically selects and runs the optimal model for your computer specifications with a few commands, requiring no complex installation process.
The Principle of Loading Experts from Disk
Traditional methods required loading the entire model into fast memory, which incurred high hardware costs. colibri leverages the characteristics of Mixture of Experts (MoE) models to keep only the experts actually used during token generation in RAM, while storing the rest on disk. The model reads from disk only when it needs a specific expert, and if a GPU is present, it prioritizes placing the most frequently used experts in GPU memory. This allows the model to function normally even if the RAM capacity is smaller than the total model size, depending on disk read speeds. This is the core technique that enables running large models on general user hardware.
Installation Starting with a Single Command
Installation proceeds via a single script on Windows, Linux, and macOS. Running ./start-here.sh or START-HERE.bat automatically analyzes your system's RAM, disk space, CPU, and GPU information. Based on this analysis, it recommends a model suitable for your specifications, building from source if a compiler is available or downloading a pre-built binary if not. If you want to stop the model download, you can pause and resume from where it left off. Once installation is complete, a browser dashboard opens, displaying an API address usable by other applications. Subsequent runs start immediately without downloading or building.
Supported Models and Hardware Requirements
Currently, 13 engines are supported, including 10 language models, 1 image generation model, and 2 decision-making models. It supports the latest open-source models such as GLM-5.2, DeepSeek V4, Qwen3.6, and Kimi K3. A minimum of 8GB RAM and 22GB of disk space is required, while running larger models benefits from 16GB or more of RAM and a fast NVMe disk. A GPU is not mandatory, as all engines run on CPU alone. It supports integrated and discrete GPUs from AMD, Intel, and NVIDIA via Vulkan, and NVIDIA cards can achieve higher performance through CUDA. Apple Silicon supports GPU operations for some engines via the Metal backend.
Dashboard and System One Features
The browser-based dashboard provides pages for chat, System One, Brain, and profiling. System One performs probability-based decision-making rather than text generation. When given a state and a question, it returns probabilities and confidence levels for allowed options, expressing when the model is uncertain in numerical terms. This is useful for binary or multiple-choice questions such as review approval and risk assessment. It maintains compatibility with the existing Jev API, allowing existing clients to switch without modification. The dashboard allows you to observe each expert activating in real time, helping to understand the model's internal operations.
Things to Check Before Running
This tool is written in pure C with no dependencies and follows a policy of not silently changing the precision or router behavior of specific models. However, because it streams experts from disk, disk read speed significantly influences response time relative to model size. If you do not have a fast NVMe drive, responses for large models may be slow. Additionally, some models have non-commercial licenses or specific conditions, so you should verify if the model suits your intended use. If using a GPU, it is recommended to check that your Vulkan or CUDA driver version is up to date and that your GPU is included in the optimization targets for the engine. The README provides accurate benchmark figures for all models, allowing you to assess performance in environments similar to your hardware.
