From text to 3D: the magic behind Edify 3D by NVIDIA

The demand for high-quality 3D assets is booming across industries like video game design, extended reality, film production, and simulation. However, crafting production-ready 3D content often involves a complex, time-intensive process requiring advanced skills and tools. Addressing these challenges is Edify 3D by NVIDIA – a solution that leverages AI technologies to make 3D asset creation faster, easier, and more accessible.

Edify 3D sets a new benchmark in 3D asset creation by enabling high-quality asset generation in under two minutes. This innovative platform produces 3D models with detailed geometry, clean mesh topologies, UV mapping, 4K resolution textures, and physically-based rendering (PBR) materials. Whether the input is a text description or a reference image, Edify 3D can generate stunningly accurate 3D assets suitable for a wide range of applications.

Compared to traditional text-to-3D generation approaches, Edify 3D not only delivers superior results in terms of detail and realism, but also outperforms in efficiency and scalability.

Edify 3D’s core technology leverages advanced neural networks, combining diffusion models and Transformers to push the boundaries of what AI can achieve in 3D asset generation. The process begins with multi-view diffusion models that synthesize the RGB appearance and surface normals of an object from different viewpoints. These multi-view images then serve as input for a Transformer-based reconstruction model that predicts the geometry, texture, and materials of the final 3D shape.

The pipeline is highly optimized for scalability, with the ability to handle both text-to-3D and image-to-3D inputs. For text-to-3D generation, users provide a natural language description, and the model synthesizes the object based on predefined prompts and poses. For image-to-3D, the system can automatically extract the foreground object from a reference image and generate its 3D counterpart, complete with unseen surface details.

To achieve its impressive results, Edify 3D relies on a meticulously designed data processing pipeline. The system begins by converting raw 3D shape data into a unified format, ensuring compatibility and consistency across datasets. Non-object-centric data, incomplete scans, and low-quality shapes are filtered out through active learning with AI classifiers and human oversight. Canonical pose alignment ensures that all shapes are properly oriented, reducing ambiguity during training.

For training purposes, Edify 3D employs photorealistic rendering techniques to generate multi-view images from the processed 3D shapes. A vision-language model is then used to generate descriptive captions for the rendered images, enriching the dataset with meaningful metadata.

For text-to-3D use cases, Edify 3D produces detailed 3D models that align perfectly with user-provided descriptions. In image-to-3D scenarios, the system accurately reconstructs the 3D structure of the reference object while “hallucinating” realistic textures for unseen areas, such as the back of an object.

Edify 3D’s outputs stand out for their exceptional quality. The generated assets include clean quad mesh topologies, sharp textures, and detailed geometry. These features make them ideal for downstream editing workflows in industries like gaming, animation, and product design.

Read more about the Scalable High-Quality 3D Asset Generation in the article on arXiv.