Compress data at over 500 MB/s per core with a lossless tool
If you need to process large amounts of data in real time but are slowed down by the compression step, this tool may be the answer. LZ4 is a lossless compression method that restores data perfectly to its original state, achieving compression speeds of over 500 MB/s per core and scaling proportionally on multi-core CPUs. Decompression also runs at several GB/s per core, often reaching the limits of memory speed.
Balancing speed and compression ratio
LZ4 does not have fixed performance; instead, it allows you to adjust the balance between speed and compression ratio depending on the situation. Dynamically selecting the 'acceleration' factor increases speed at the cost of a lower compression ratio. Conversely, if you want a higher compression ratio, you can use LZ4_HC, a derived version that improves the ratio by using more CPU time. Regardless of the version used, the decompression speed remains the same, minimizing latency in the data restoration step regardless of the compression method.
Improving small file performance with dictionary-based compression
LZ4 supports dictionary-based compression in both its API and command-line interface (-D). Although it uses only the last 64KB of the input file as the dictionary, combining this with the Zstandard dictionary builder can significantly improve compression performance for small files. This feature is particularly useful for efficiently handling small files with many repeating patterns.
Benchmark results and comparisons
The benchmarks presented in the README were generated using lzbench v2.4.1 on a Core i7-9700K CPU, compressing the Silesia Corpus in single-thread mode. LZ4 default settings (v1.11.0) recorded a compression speed of 770 MB/s and a decompression speed of 4950 MB/s. Under the same conditions, LZO 2.10 showed a compression speed of 660 MB/s, Snappy 1.3.1 showed 520 MB/s, and Zstandard 1.5.7 showed 510 MB/s. In terms of decompression speed, LZ4 is much faster at 4950 MB/s compared to LZO (800 MB/s) or Snappy (1350 MB/s). The high-compression mode LZ4 HC -9 has a compression speed of 40 MB/s and a decompression speed of 4800 MB/s, offering a higher compression ratio but slower compression speed than the default mode.
Installation and build system
LZ4 is written in C and is built with the make command and installed with make install. This installation command may require root privileges. The Makefile follows standard conventions and supports parallel builds (-j#). Other build systems, such as CMake, Meson, and Visual Studio, are also provided in the build/ directory. It can also be installed via the vcpkg dependency manager, a port maintained by the Microsoft team and community contributors to keep it up to date.
License and pre-use considerations
The LZ4 library is provided under the BSD 2-Clause license, while the lz4 command-line utility is provided under the GPL-2.0-or-later license. You must verify license compatibility according to the nature of your project. In addition to the C reference source, versions ported to several languages, including Java, C#, Python, Perl, and Ruby, exist, and this list is managed on the LZ4 homepage. Most distributions support easy installation of the liblz4 library and the lz4 command-line interface through package managers. The compression format is defined in block format and frame format, and the frame format must be adhered to for interoperability.
