Run trained AI models directly on Intel hardware
You can run models built with PyTorch or TensorFlow directly on Intel CPUs, GPUs, and NPUs without a separate conversion process. Inference is possible without installing the original framework, which simplifies the deployment environment. You can start the installation with the pip install -U openvino command.
Support for various framework models
It supports models trained with PyTorch, TensorFlow, ONNX, Keras, PaddlePaddle, and JAX/Flax. Transformers and diffusers models from the Hugging Face Hub can be integrated directly through Optimum Intel. You can convert and deploy models without the original framework, simplifying the deployment environment. It provides C++, Python, C, and NodeJS APIs, along with dedicated APIs for generative AI.
Supported hardware and platforms
It allows for efficient deployment with reduced resource requirements across various platforms, from edge devices to the cloud. It supports inference on CPUs with x86 and ARM architectures, Intel integrated and discrete GPUs, and Intel NPUs. You can select the optimal available device to run models without being dependent on specific hardware.
Utilizing generative AI and LLMs
It provides dedicated tools and samples for large language model (LLM) and generative AI workflows. You can learn how to optimize and run generative AI models through various examples, such as LLM-based chatbots, text-to-image generation, multimodal assistants, and speech recognition. You can build LLM inference pipelines and improve performance through the OpenVINO GenAI repository and Jupyter notebooks.
Ecosystem and integration tools
Various tools and framework integrations are available to help with model optimization and deployment. You can apply advanced optimization techniques such as quantization and sparsity through NNCF, and implement scalable model serving through OVMS. It connects easily with existing ecosystems like vLLM, ONNX Runtime, LangChain, LlamaIndex, and Keras 3, allowing you to add an OpenVINO backend to existing codebases.
Pre-adoption considerations
The license is specified as Apache License Version 2.0, allowing for commercial use. Since it is optimized for Intel CPUs, GPUs, and NPUs, it is difficult to expect performance benefits on hardware from other manufacturers. Telemetry data collection is enabled by default, so if you do not want this, you must disable it with the opt_in_out --opt_out command. It is recommended to check system requirements and supported devices in advance and plan your hardware configuration by referring to performance benchmark data.