Run specific large language models on consumer hardware with an optimized inference engine
Imagine running large language models directly on a MacBook with 128GB of RAM or a DGX Spark. This tool is an inference engine optimized for specific models, allowing you to run models, chat, or handle coding tasks without complex configuration. Its goal is to drive models stably even on uncommon hardware combinations.
Optimization for specific models rather than universality
This project is not a universal tool that supports all models; it focuses on a few high-quality models such as DeepSeek V4 Flash and PRO, GLM 5.2/5.3, and Qwen3.8 Flash Next. You must use only the GGUF files generated by the project itself, as it does not support files converted externally. This limitation is the core strategy for maximizing performance and stability in specific hardware environments. Model support is maintained flexibly, and existing models may be removed if better alternatives emerge.
Execution methods and performance by hardware
Metal-based Apple Silicon is the primary target, delivering optimal performance on Macs with 96GB or more of RAM. In environments with insufficient RAM, models can be run via SSD streaming. In NVIDIA CUDA environments, the DGX Spark is the main target, and it also supports multi-GPU systems not supported by other backends. For example, a configuration of eight L40S cards with the Ada Lovelace architecture achieved a generation speed of approximately 126 tokens per second across 16 sessions. It also supports ROCm-based Strix Halo systems.
Coding agent and server features
The built-in ds4-agent performs inference directly without a separate HTTP server, managing token history and model state together. Sessions are stored in ~/.ds4/kvcache and can be resumed later, though sessions containing images are not yet saved. Using ds4-server allows integration with various clients such as Pi, OpenCode, Codex CLI, and Claude Code. The server runs on port 8000 of localhost by default and can handle multiple sessions simultaneously.
Pre-use checks
This software is of beta quality, and instability or regression issues may occur with version updates. Although developed with the help of AI coding agents, humans led the ideas and testing. Because hardware requirements are strict, you should first check if your system has 96GB or more of RAM or if SSD streaming is possible. You should also refer to the model guide to ensure that the quantization format and memory size of a specific model match your environment. The license is MIT, and it explicitly acknowledges contributions from llama.cpp and GGML.
