Run and fine-tune large language models on consumer hardware
This project sets up an environment to run and train models with billions of parameters without datacenter-grade GPUs. It applies heterogeneous computing by using CPU and GPU together, placing parts of the model in memory and parts on the graphics card. This allows complex AI tasks to be handled with limited hardware resources.
Recent updates and research outcomes
A preprint of KTransformers-FineTune (KT-FT) was recently released, presenting layout-aware and scheduling optimization techniques for CPU-GPU co-computation in MoE model fine-tuning. In August 2026, an MoE fine-tuning cookbook integrated with LlamaFactory was added, guiding the process from hardware checks to troubleshooting.
Two axes: inference and fine-tuning
This repository provides two functions, Inference and SFT (fine-tuning), from the kt-kernel source tree. The inference function performs INT4/INT8 quantized inference using CPU-optimized kernels that leverage Intel AMX and AVX512/AVX2, applying NUMA-aware memory management. The fine-tuning function integrates with LlamaFactory to enable training of large MoE models like DeepSeek-V3/R1 with limited GPU memory. Specifically, it shows a 6 to 12 times improvement in training speed in benchmarked MoE SFT workloads compared to ZeRO-Offload.
Performance and hardware configuration
As an example of inference performance, the DeepSeek-R1-0528 (FP8) model recorded a total of 227.85 tokens/s and an output of 87.58 tokens/s (8-way concurrency) on a combination of 8 L20 GPUs and a Xeon Gold 6454S CPU. For fine-tuning, DeepSeek-V3 and R1 models can be trained at a speed of 3.7 iterations per second (it/s) using approximately 80GB of total GPU memory across 4 RTX 4090s. The Qwen3-30B-A3B model achieves a speed of over 8 iterations per second using approximately 24GB of memory on a single RTX 4090. CPU memory usage has been reduced to about half compared to the previous KT SFT path in benchmark settings.
Installation and getting started
To use the inference function, move to the kt-kernel directory and run the pip install . command. For fine-tuning, install LLaMA-Factory and then install the ktransformers[sft]==0.7.0 and sglang-kt==0.7.0 packages. Afterward, you can start training by specifying the FSDP2 KT INT8 configuration file via the accelerate launch command. Python is the primary language, and integration with SGLang allows application to production serving environments.
Target audience and verification items
This tool is suitable for researchers and developers who want to experiment with or train large MoE models without high-cost GPU infrastructure. It is particularly advantageous for users familiar with heterogeneous computing environments that mix CPU and GPU usage. Before use, verify that the hardware supports Intel AMX or AVX512/AVX2, and check compatibility with LlamaFactory. It is also important to check whether the support scope for specific models (DeepSeek, Kimi, GLM, etc.) and quantization formats (INT4/INT8/FP8) match your workload. The README does not specify license details or long-term support scope, so additional verification is required before commercial use for purposes other than research.
