Run a 27B language model for images and tools on your local machine
Imagine uploading photos or PDFs in a browser, executing code, or performing web searches during a conversation. This repository enables you to handle such interactions on your own device without relying on external servers. It is a demo that sets up the Bonsai 2 model and its runtime environment created by PrismML in one go.
Model size selection and compression efficiency
Bonsai 2 is available in four sizes: 2B, 4B, 9B, and 27B. The 27B model is the default, but you can choose a smaller model to reduce the load on your local environment if your hardware specifications are low. The 27B model is compressed to approximately one-ninth the size of FP16 while retaining 98.2% of its intelligence, with no significant difference in math and coding performance compared to full precision. Each size is provided in both GGUF and MLX formats, making it usable on Mac and other platforms.
Vision, tool calling, and reasoning capabilities
This model supports vision capabilities that allow it to accept photos, screenshots, and PDFs to answer questions about their content. It also natively supports OpenAI-style tool calling, enabling integration with external tools via MCP servers. Designed as a reasoning model, it allows you to adjust the reasoning effort for each conversation and efficiently processes a long context of 262K tokens using a hybrid attention structure.
Installation and server execution
On macOS and Linux, running ./setup.sh automatically installs system dependencies, the Python environment, model weights, and the llama.cpp binary. On Windows, you can run setup.ps1 in PowerShell. Once installation is complete, run ./scripts/start_llama_server.sh to start the local server and access http://localhost:8080 in your browser to immediately use chat, vision, and tool features. For Apple Silicon Macs, it is recommended to use ./scripts/run_mlx.sh or ./scripts/start_mlx_server.sh for the MLX backend.
Options based on hardware and format
llama.cpp automatically detects various hardware such as Metal, CUDA, Vulkan, ROCm, and CPU to perform optimal GPU offloading. The PQ2_0 format uses an additional 1.3GB of memory to increase prompt processing speed and is the default download target for this demo. The PTQ1_0 format offers a smaller size with 1.75-bit weights but requires the PrismML fork version of llama.cpp. The official Q2_0 format works with the latest upstream llama.cpp, but certain experimental features may require the fork version.
Things to check before running
The PQ2_0 weights for the 27B model are approximately 7.8GB, and additional capacity is required if you include Open WebUI and the code interpreter. In environments with limited memory, set BONSAI_OPENWEBUI=0 and BONSAI_CODE_INTERPRETER=0 to skip unnecessary components. Since it is a reasoning model, it goes through a thinking process before generating a response, so setting a generous output limit or lowering the reasoning effort to 'medium' can help improve speed. Apple Silicon users should ensure the MLX environment is properly set up and terminate any existing MLX server before running the start script.
