Compress models to boost inference speed and prepare them for deployment
You can handle the process of reducing model size and increasing speed in one place before putting a trained model into a production environment. It combines various optimization techniques such as quantization, pruning, and distillation to compress the model, then exports it in a format that deployment frameworks can use immediately. As a result, memory usage decreases and response times improve, creating the possibility to run larger models on limited hardware.
Recent announcements and key changes
Recently, a W4A4 NVFP4 quantization tutorial for the Qwen3.6-35B-A3B model was added, demonstrating how to achieve 1.30x higher throughput and a 3.1x smaller checkpoint compared to BF16 while recovering accuracy. Additionally, a case study was published showing that quantizing the Nemotron 3 Ultra 550B model to NVFP4 achieves up to 5.9x higher decode-centric inference throughput compared to GLM-5.1 754B FP4 while maintaining BF16-level accuracy. In December 2025, the official name was changed from 'NVIDIA TensorRT Model Optimizer' to 'NVIDIA Model Optimizer', and in January 2025, it transitioned to open source to accept community contributions.
Techniques for model compression and speed improvement
This tool goes beyond simply converting models, allowing you to combine various optimization techniques via a Python API.
- Post-training quantization Compresses model size by 2 to 4 times to increase inference speed while maintaining quality.
- Pruning and distillation Removes unnecessary weights or trains a smaller model to behave like a larger one to reduce deployment size.
- Speculative decoding Trains a draft module to predict additional tokens during inference, reducing latency.
Input formats and deployment framework integration
It accepts models in Hugging Face, PyTorch, and ONNX formats as input to perform optimization tasks. Optimized quantized checkpoints are designed to be applied directly to downstream deployment frameworks such as SGLang, TensorRT-LLM, TensorRT, and vLLM. The Hugging Face export API supports both transformer and diffuser models, and it integrates with NVIDIA Megatron-Bridge, Megatron-LM, and Hugging Face Accelerate to support the training process for required inference optimization techniques. This allows developers to quickly deploy optimized models to production environments without directly constructing complex deployment pipelines.
Installation methods and requirements
The stable release package can be installed from PyPI using the pip install -U nvidia-modelopt[all] command, during which additional third-party open-source software is downloaded. To install with development dependencies from source or to use the latest features, clone the repository and run pip install -e .[dev]. Using NVIDIA container images (nvcr.io/nvidia/pytorch, nvcr.io/nvidia/nemo, nvcr.io/nvidia/tensorrt-llm/release) allows you to start working immediately in an environment where Model Optimizer is pre-installed. Before installation, you should review the license terms of the included open-source projects.
Pre-use checks and precautions
This tool is still in a pre-1.0 state, so the migration period for deprecated features is set to a short duration of approximately one month (one release). Therefore, specific APIs or parameters may be removed, so it is recommended to check the change log and upgrade to the latest version before applying it to production environments. Supported model types and techniques are divided into LLM/VLM quantization, diffuser quantization, ONNX quantization, Windows quantization, QAT, pruning, distillation, and speculative decoding, and you must check the support matrix for each item. Additionally, because this tool is tightly integrated with the NVIDIA AI software ecosystem, some features may be limited in environments outside that ecosystem.
