Fine-tune 8B language models on 4GB laptop GPUs with a single command
You can train large language models directly on a laptop with integrated graphics, without cloud servers or complex remote access setups. Soup is a tool that handles the entire fine-tuning process with a single YAML configuration file and one command. This allows developers to focus on improving the model itself rather than spending time on infrastructure setup.
v0.75.0: No longer ignoring configuration errors
The recently released v0.75.0 version immediately stops execution if it finds a key in the configuration file that does not exist. In previous versions, typos or unsupported options were silently ignored and training proceeded, risking unintended results. Now, both the CLI and API accurately report incorrect field names, and the MLX backend verifies that configuration values are actually applied. Additionally, validation loss is now displayed in real time on the dashboard, providing a clearer view of the training status.
Overcoming VRAM limits with layer streaming
Soup's core feature, layer streaming, works by passing decoder layers one by one from host RAM to the GPU in sequence, without loading the frozen base model into GPU memory. This allows training the Llama-3.1-8B-Instruct model with NF4 quantization even in limited environments like an RTX 3050 Laptop 4GB. Measurements show peak memory usage of 3.32GB and a processing speed of 119.6 tok/s. This method guarantees bit-for-bit identical results to standard execution and was reproduced with the same memory usage on an H100. However, this feature is still in beta and must be explicitly enabled with the stream_layers: true option.
The flow from installation to deployment
Installation is completed with a single line: pipx install "soup-cli[train]". You can also manage experiments and view real-time metrics from a browser via the web UI (soup ui). Data formats such as Alpaca, ShareGPT, and ChatML are auto-detected, so you only need to specify the JSONL file path.
Hardware and version checks before use
Soup supports Python 3.10 through 3.12, while versions 3.13 and above are not yet verified. An NVIDIA card with CUDA support is recommended for the GPU, and Apple Silicon (MPS) and CPU are also supported, though CPU is experimental and very slow. A minimum of 8GB VRAM is required for QLoRA training of 7B models. The license is Apache-2.0 and free. However, since the layer streaming feature is in beta, it is recommended to run comparison tests with existing methods for important projects.
