Edit sheet music to produce finished songs
Enter lyrics and a style description, and the system first generates sheet music containing melody and harmony. This sheet music is then converted into a finished song with vocals and accompaniment. Because you can review and modify the generated sheet music directly, you can refine the composition to your liking before hearing the final audio. YuE2 is a tool that visualizes the music generation process as sheet music, allowing for direct intervention.
Plan with sheet music, finish with audio
YuE2 begins by taking text input and planning the structure of the melody and harmony. This plan is output in ABC notation, which users can read or edit directly. When the modified sheet music is re-entered, the system generates a finished song with vocals and accompaniment based on that structure. This approach allows you to understand and intervene in the structure of the generated music.
Covers and interactive editing
A zero-shot cover feature converts existing audio into sheet music and reinterprets it in a new style. The system also supports an editing workflow where you converse with an agent to modify harmony, melody, and lyrics to generate new audio. All these processes use the same model checkpoint, enabling various musical variations without separate fine-tuning.
WildSongBench evaluation results
In the WildSongBench evaluation, YuE2 was tested on 192 prompts. With a best-of-8 setting, it recorded an average SongBench score of 6.9632, achieving the highest average score among all evaluated settings. This score is competitive with major commercial systems such as Suno v5/v6. However, note that the differences between the top scores do not establish statistical significance.
Runtime environment and license
To run locally, you need a Linux environment, Python 3.12, and an NVIDIA GPU with at least 24GB of VRAM that supports BF16. Model weights are downloaded from Hugging Face, and the code is under the Apache 2.0 license. Model weights are under the CC BY-NC 4.0 license with additional creator rights, allowing free use and monetization for individuals and content creators. Commercial licenses for enterprises require separate negotiation, so you must verify the license terms before any commercial use.
