Run and fine-tune 70B language models on a 4GB graphics card
You can run large language models directly on your computer without high-end servers. AirLLM saves memory by loading model layers one at a time, enabling inference for large models on small graphics cards. Recent updates added support for fine-tuning, allowing model training experiments in limited hardware environments.
Overcoming memory barriers with layer streaming
Previous methods required loading the entire model into graphics card memory, but AirLLM loads only one layer to the GPU at a time while keeping the rest on disk. This means memory usage is determined by the size of a single layer rather than the total model size. As a result, 70B models run on 4GB and 671B models on approximately 12GB of VRAM. The core benefit is significantly reduced memory load while maintaining the performance of the original model without precision-loss processes like quantization or distillation.
Latest updates for inference and training
Frozen base weights are streamed from disk at the layer level, while only adapters reside on the GPU. This design allows the Qwen3.8-Flash-Next(125B) model to be trained with less than 6GB of VRAM, and Qwen3.8-27B can be trained with approximately 2GB at a sequence length of 512. Additionally, official support for Qwen3.8-Flash-Next and Qwen3.8-27B models was added in August 2026, enabling models with the latest architectures to be used through the same interface.
VRAM usage and support scope by model
AirLLM supports various open-source models using Hugging Face's AutoModel interface.
- Small and medium models (8B~30B) 8B-class models such as Qwen3, Mistral, and Phi use approximately 1~2GB, while 30B-class MoE models use approximately 1~3GB of VRAM.
- Large models (70B~235B) Llama 3.x 70B runs on approximately 4GB, Qwen3-235B on approximately 3GB, and Qwen3.8-27B on 3.33GB of VRAM.
- Extra-large models (405B~671B) Llama 3.1 405B operates on approximately 8GB, DeepSeek-V3(671B) on approximately 12GB, and Kimi K3(2.8T) on 3.72GB of VRAM.
Preparation for installation and execution
AirLLM is installed with the pip install airllm command and supports Apple Silicon chips on macOS. Since models are stored split by layer during inference, sufficient disk space is required in the Hugging Face cache directory. Specific models require additional dependencies: Kimi K3 needs compressed-tensors and flash-attn installation along with CUDA 12-based torch, while Qwen3.8-Flash-Next requires a specific build of transformers. If disk space is insufficient during the model splitting process, the delete_original option can be used to delete original files to free up space.
Technical constraints to check before use
AirLLM relies on disk I/O to maximize memory efficiency, so fast storage devices like SSDs affect inference speed. Using model compression (4bit/8bit) features can increase inference speed by up to 3 times, but this requires installing the bitsandbytes library and airllm version 2.0.0 or higher. The training feature operates differently from Hugging Face Trainer or bitsandbytes QLoRA, and for certain models, installing the latest development version of the transformers library is mandatory. It is suitable for developers who want to experimentally handle large models in hardware-limited environments, but caution is needed for services where real-time performance is extremely critical, as disk loading delays may occur.
