Run machine learning tensor operations anywhere with dependency-free C/C++
Get a lightweight tensor library that builds immediately with just a compiler. You can run the same code on various hardware, from x86 to ARM and RISC-V, without complex installation steps. This low-level tool handles core operations for machine learning models, providing an environment where you have direct control without being tied to a specific framework.
Lightweight and portable design principles
The goal of this library is to provide simple, portable, and efficient tensor operations. As a pure C/C++ implementation with no external dependencies, it can be built on any platform without installing additional libraries. No memory allocation occurs during execution, maintaining predictable performance. It supports a wide range of architectures, including x86, ARM, RISC-V, LoongArch, PowerPC, s390x, and WebAssembly, with SIMD-optimized kernels applied to x86, ARM, and RISC-V.
Support for diverse hardware and precision
It is designed to operate on various backends, including CPUs, GPUs, NPUs, and browsers. It supports integer quantization from 2-bit to 8-bit, as well as MXFP4 and NVFP4 microscaling formats. This is a key feature that enables model inference even in memory-constrained environments. The examples/simple folder contains a minimally commented example demonstrating matrix multiplication, allowing you to quickly grasp the basic functionality.
Building and contributing
To build from source, clone the repository using git clone, then create a build directory and compile using the cmake command. When building with the release configuration, you can perform parallel compilation using the -j 8 option. If you have changes to the core library or the CMake build system, it is recommended to open a PR in the llama.cpp repository. This increases the likelihood that the PR will be better exposed, tested, and reviewed.
Choose it when you need direct control
This tool is suitable when you need to manage the details of tensor operations directly without using high-level frameworks. It is useful for developing custom operators optimized for specific hardware or in embedded environments where memory usage must be extremely limited. However, since this library itself does not provide an abstracted API, you need the ability to write and manage code directly. The README does not specify the license or long-term support scope, so you should verify these details separately before applying it to your project.