OpenAI announces Point-E — a system for generating 3D objects

The San Francisco-based open Artificial Intelligence platform OpenAI announced the release of Point-E — a machine-learning system that allows users to generate a 3D object based on a simple text input.

A team of researchers has developed a completely new approach. Point-E doesn’t create 3D objects in the traditional sense. Instead, it creates point clouds, or discrete sets of data points in space that represent a three-dimensional shape.

Generating point clouds is far easier than generating real images, but they don’t capture an object’s fine-grained shape or texture — a key limitation of Point-E currently. To get around this limitation, the Point-E team trained an additional AI system to convert point clouds to meshes.

Point-E consists of two models: a text-to-image model and an image-to-3D model. The text-to-image model, similar to generative art systems like OpenAI’s own DALL-E 2, was trained on labeled images to understand the associations between words and visual concepts. The image-to-3D model, on the other hand, was given a set of images paired with 3D objects to learn how to effectively translate between the two of them.

One of the biggest advantages of this approach is that it is very fast and undemanding in terms of hardware required to produce the final image.

The OpenAI researchers note that Point-E’s point clouds could be used to fabricate real-world objects, such as through 3D printing. With the additional mesh-converting model, the system could also find its way into game and animation development workflows.

“We find that Point·E is capable of efficiently producing diverse and complex 3D shapes conditioned on text prompts. We hope that our approach can serve as a starting point for further work in the field of text-to-3D synthesis”, — said the researchers.

Learn more about Point·E in the paper

The code is available on GitHub