Achieve real-time compression speeds with high compression ratios
There are moments when you need to compress data quickly while significantly reducing its size. Zstandard is an algorithm designed to achieve higher compression ratios than existing standards while targeting real-time processing. This repository provides the official implementation of the algorithm, offering a C library and command-line tools.
The balance between speed and compression ratio
Zstandard secures performance through a fast entropy coding step based on the Huff0 and FSE libraries. In tests using the Silesia corpus on a Core i7-9700K CPU, the default setting (-1) recorded a compression speed of 510MB per second and a compression ratio of 2.896. Additionally, the --fast option can increase speed further; the --fast=4 setting shows a compression speed of 665MB per second and a compression ratio of 2.146.
Pre-training for small data processing
To address the issue of decreasing compression efficiency with smaller data volumes, a pre-training mode is provided. By providing samples of a specific data type, the algorithm learns from them to generate a 'dictionary' file. Loading this dictionary file during compression and decompression significantly improves the compression ratio, particularly in the first few KB. It is most effective to generate and use separate dictionaries for each data type.
Support for various build systems
This repository uses make as its default build system and is compatible with various build tools such as cmake, Meson, VCPKG, and Conan. Running only the make command generates the zstd command-line utility in the root directory and the libzstd library in the lib/ folder. In Windows environments, it is recommended to either open the Visual Studio project directly or generate a solution via cmake. Universal2 builds supporting both Apple Silicon and Intel architectures are also possible through the CMAKE_OSX_ARCHITECTURES option in cmake.
Stability and security verification
The Zstandard file format is documented in RFC8878, and multiple independent implementations exist. This repository is the official reference implementation, released under a dual BSD or GPLv2 license. It is used to compress large amounts of data in Meta and other large-scale cloud infrastructures, and it is continuously checked for security vulnerabilities through Google's oss-fuzz program. The dev branch is where all contributions are merged, and direct commits to the release branch are not allowed.