Run local model inference faster by auto-tuning kernels to your hardware
The moment you download a model, it automatically finds and applies the calculation method that best fits your hardware. This allows the same model to run faster than existing tools. It is an open-source inference engine that adjusts kernels to match your hardware.
Automatic optimization for hardware
Existing tools apply the same kernels to all hardware, but Magnitude performs direct tests on your chip when you download a model. It tries various combinations, such as GPU thread count, data chunk size, and pre-load distance, for up to one minute to select the fastest method. The model's output results do not change during this process; it focuses solely on increasing calculation speed. Thanks to this automatic tuning, it achieves an average speed of more than 2x compared to Ollama on Mac and NVIDIA environments.
Agent connection and memory management
Connect to existing agents like Pi, OpenCode, Hermes, and Codex with a single click.
- Built-in draft model It provides the fastest speculative decoding drafter for each model to increase response speed.
- Complete privacy Prompts, files, and models all stay on your machine, and it works without an internet connection after the model is downloaded.
Installation and server deployment
It offers a desktop app and CLI on macOS, Linux, and Windows.
- Supported platforms It works in environments with Apple Silicon, NVIDIA, AMD, or Intel GPUs, or CPU-only setups.
- Operating systems It supports macOS, Linux, and Windows, with Linux available for both x64 and ARM64 architectures.
Things to check before applying
This tool ensures performance by directly writing optimized kernels for popular open-weight model families. Therefore, you must check the list of supported models at magnitude.dev/models, as not all models receive the same level of optimization. There is no fixed minimum hardware specification, but running larger models requires more memory. It is distributed under the Apache 2.0 license and is suitable for users who prioritize zero token costs and ensuring data does not leave their system.
