Generate complex 3D assets with materials from a single image
This tool creates detailed 3D models from a single image, handling open surfaces like fabric or leaves and complex hollow structures. It addresses common issues in 3D generation, such as shape distortion and material loss, using a new data structure. The output is immediately convertible into 3D assets with PBR materials ready for rendering.
A new 3D representation with O-Voxel
Existing 3D generation models use iso-surface fields to represent shapes, but this approach loses information when handling complex structures like open surfaces or non-manifold geometry. To overcome this, the repository introduces a new sparse voxel structure called 'O-Voxel'. This structure encodes 3D assets into a compressed latent space, preserving complex topology and sharp features. As a result, shapes that were difficult to reproduce with previous methods, such as clothing or plants, can now be generated with precision.
Generation quality and processing speed
This speed is the total time required, combining shape generation and material generation times. Additionally, the data processing pipeline is simplified, taking less than 10 seconds on CPU to convert a textured mesh to O-Voxel and less than 100 milliseconds on CUDA to convert O-Voxel back to a mesh. This efficiency makes it suitable for real-time or near-real-time 3D content production workflows.
PBR materials and transparency support
The model surfaces various properties beyond simple color, including Base Color, Roughness, Metallic, and Opacity. This enables photorealistic rendering and supports transparency effects. Generated 3D assets can be exported in GLB format, with the default setting being OPAQUE mode. To enable transparency, you must manually connect the alpha channel of the texture to the material's opacity input in 3D software. These material properties are generated directly by the model without separate post-processing, reducing work time for 3D artists.
Execution environment and installation requirements
This code has currently been tested only on Linux systems. Hardware requirements include an NVIDIA GPU with at least 24GB of memory, verified on NVIDIA A100 and H100. For software, CUDA Toolkit version 12.4 is recommended, and it is advised to manage dependencies using Conda. The Python version must be 3.8 or higher. The installation process is handled via the setup.sh script, which installs several specialized packages including flash-attn, nvdiffrast, and nvdiffrec. If the GPU does not support flash-attn (e.g., NVIDIA V100), you must manually install xformers and set the environment variables.
Considerations before application
Although this repository is released under the MIT license, some dependencies like nvdiffrast and nvdiffrec follow their own separate license terms. Therefore, you must verify the license terms of these dependencies before applying them to commercial projects. Currently, inference code and pre-trained weights (4B) are provided, and training code is also publicly available, allowing fine-tuning with custom datasets. However, generating complex 3D structures requires significant GPU memory and computing resources, so it is recommended to check if your hardware specifications meet the requirements before use.
