Run and serve large language models with an open-source engine
Imagine a scenario where response speed is maintained even when servers are flooded with requests, and more models can run simultaneously without wasting memory. vLLM is an open-source library for the inference and serving of large language models. It allows you to run various Hugging Face models immediately without complex hardware configurations.
Memory management and throughput optimization
vLLM efficiently allocates attention key and value memory through PagedAttention technology. It reduces memory waste, similar to how an operating system manages virtual memory, allowing it to process more requests. It also supports continuous batching, which keeps bundling incoming requests into batches for processing, and prefix caching, which reuses already calculated prompt prefixes. These features combine to produce high throughput.
Hardware and model architecture support
It supports a wide range of hardware, including NVIDIA and AMD GPUs, Intel GPUs, and x86, ARM, and PowerPC CPUs. Various hardware plugins are also available, such as Google TPU, Intel Gaudi, and Apple Silicon. On the model side, it supports over 200 Hugging Face model architectures, including Llama, Qwen, DeepSeek-V3, and GPT-OSS. It handles a broad spectrum of models, from decoder-only models to mixture-of-experts models, multimodal models, and embedding models, and can apply various quantization methods such as FP8, INT8, GPTQ, and AWQ.
APIs and features for immediate practical use
It provides an OpenAI-compatible API server, allowing you to switch with almost no modifications to existing OpenAI client code. It also supports the Anthropic Messages API and gRPC. It has built-in features necessary for actual services, such as streaming output, structured output generation, and tool call parsing. It can perform distributed inference through tensor, pipeline, data, expert, and context parallelism, and can efficiently process multiple LoRA adapters simultaneously.
Installation and getting started
Installation is simple via uv or pip. After installing with the uv pip install vllm command, running vllm download-kernels fetches pre-compiled kernels to shorten startup time. Building from source is also possible for development purposes. You can find installation guides and quick start tutorials on the official documentation site vllm.ai.
Things to check before use
vLLM started at the UC Berkeley Sky Computing Lab and is an active open-source project with over 2,000 contributors. If used for research purposes, the PagedAttention-related paper must be cited. Use GitHub Issues for technical questions or feature requests, the vLLM forum for discussions between users, and Slack for contribution coordination and development-related matters. Security vulnerability reports are handled through the Security Advisories feature on GitHub. You must always check the supported model list in the official documentation for the scope of support for specific hardware or model architectures.
