Reduce first response times for long-context LLM services by reusing KV caches
When processing long conversations or document analysis, you can shorten the time until the first token appears by reusing data already calculated in previous requests. LMCache is a management layer that stores this data persistently rather than in temporary memory and shares it across multiple engines. This reduces the burden of repetitive calculations and lowers inference costs.
Recent changes and ecosystem expansion
In January, the CPU memory sharing feature between multiple nodes was promoted from experimental to production level. In October 2025, the project joined the PyTorch Foundation, solidifying its position within the ecosystem.
Hierarchical storage and non-prefix reuse
It moves KV caches from GPU memory to CPU memory, local storage, and remote backends for reuse. This hierarchical structure allows data to be shared across requests, sessions, and engine instances. Specifically, for workloads handling long contexts such as RAG or multi-turn conversations, repetitive prefill calculations are reduced, improving time to first token (TTFT). It also supports non-prefix reuse, allowing cached KV blocks to be reused from any position in the prompt.
Engine-independent deployment and observability
LMCache runs as an independent daemon process, separated from the inference engine. As a result, KV cache data is not lost even if the inference engine crashes. It provides health monitoring and performance diagnostic metrics suitable for Kubernetes environments, allowing you to track cache hit rates and lifecycles at the request and token level. These observability features help understand system status and diagnose issues in production environments.
Pluggable storage and transfer backends
A unified interface allows for easy integration of various storage and transfer backends. It supports CPU RAM, local SSD, Redis/Valkey, Mooncake, InfiniStore, S3-compatible object storage, NIXL, GDS, and more. It also provides PD separation and transfer functionality, sending KV caches from prefill workers to decode workers via NVLink, RDMA, or TCP. Researchers can also write KV transformation logic directly, such as compression, token dropping, and custom serialization, through the SERDE interface.
Pre-deployment considerations
LMCache is distributed under the Apache License 2.0 and is compatible with major open-source serving engines like vLLM. You can get started easily with the pip install lmcache command. It is hardware-vendor neutral, so it can be used in various environments including AMD, NVIDIA, Arm, and Ascend. However, it is recommended to benchmark against actual workloads to confirm specific performance differences with certain hardware or storage backends. Before production deployment, you should monitor cache hit rates and system load using observability metrics.