Generate synchronized audio-video locally and fine-tune models
Imagine typing text and getting a video where a person's lip movements and voice match naturally. This repository is the official Python package for LTX-2 model inference and LoRA training. It lets you run the video generation pipeline directly in a local environment without complex server setup and fine-tune the model to your desired style.
Video and audio handled by one model
LTX-2 is an audio-video foundation model based on the DiT architecture. It combines synchronized audio and video, various performance modes, and API accessibility into a single model. This repository provides the code needed to infer the model in Python and fine-tune it via LoRA. Users can download the model and configure the pipeline to generate their desired video and audio.
Two paths for choosing speed and quality
Model execution methods are divided into two types depending on the purpose.
- DistilledPipeline The fastest text/image-to-video conversion path. It operates quickly with 1 step of 8 steps and 2 steps of 4 steps, making it suitable for initial testing.
- DFRPipeline The path for production quality. It applies detailing IC-LoRA and spatial detailing paths, and also supports 2x or 4x fps upscaling.
Model configuration for selecting only what is needed
The LTX-2.5 model has files separated by component, allowing you to download only the parts you need.
- Transformer The distilled version is used for fast inference, while the full development model is used in the guided 2-step pipeline.
- VAE There are VAEs that process video and audio separately, and the video VAE can be chosen between a diffusion decoder and a convolution decoder.
- Upscaler The spatial upscaler is used to increase resolution, while the temporal upscaler is required when running the temporal refinement round of the DFR pipeline.
Execution environment and optimization
The repository uses uv to manage dependencies. Installing the natten additional package in Linux and CUDA environments makes diffusion video VAE decoding the fastest. On Windows and macOS, it automatically falls back to Triton or eager implementation, allowing execution with the same commands. On data center Blackwell GPUs (B200), you can manually install FlashAttention 4 to optimize performance, while Hopper GPUs use the FlashAttention 3 wheel. Other CUDA GPUs automatically use PyTorch SDPA. When writing prompts, it is best to describe the scenario specifically and in chronological order, as if a director were explaining a shot list.
Pre-execution checks
Before using this repository, you must check hardware specifications and model capacity. The model files alone require approximately 66 GiB, and running the DFR pipeline requires more VRAM. In environments with limited GPU memory, it is recommended to use the --quantization fp8-cast option or not disable memory cleaning. Additionally, you must accept the model conditions on Hugging Face and log in with a read-access token to download the model. LoRAs are only compatible with the trained model, so LTX-2.3 and LTX-2.5 checkpoints cannot be exchanged with each other.
