NVIDIA's Neuralangelo AI Tool To Turn 2D Videos Into 3D Scenes Is Now Source Available
NVIDIA Research has built and released an open-source AI model that can reconstruct 2D videos into detailed 3D structures, complete with intricate details and textures. Creators can then take these 3D objects and inject them into various design applications for further manipulation, be it a video game asset, robotics, or whatever else.
The open-source tool is called Neuralangelo, which as you might have guessed is an homage to Michelangelo di Lodovico Buonarroti Simoni, the Italian Renaissance sculptor, painter, architect, and poet who is widely considered one of the greatest artists to have ever lived. Beyond the namesake, though, Neuralangelo is based on NVIDIA's Instant NeRF rendering technology, which turns 2D images into 3D scenes.
In similar fashion, Neuralangelo adopts instant neural graphics primatives to capture finer details of complex images, such as roof shingles, panes of glass, and smooth marble, and then generates high fidelity 3D models. The upshot for creative professionals is that they can record a scene using their smartphone and then have easy access to usable virtual objects.
"The 3D reconstruction capabilities Neuralangelo offers will be a huge benefit to creators, helping them recreate the real world in the digital world," said Ming-Yu Liu, senior director of research and co-author on the paper. "This tool will eventually enable developers to import detailed objects—whether small statues or massive buildings—into virtual environments for video games or industrial digital twins."
At a high level, how it works is Neuralangelo selects several frames from an object captured in various angles in a 2D video, similar to how a sculptor looks at different sides of their source material. This proves a sense of depth and size.
After the AI tool pinpoints the camera position of each frame, it gets busy creating rough 3D models of the scene. NVIDIA likens this to a sculptor making their first pass with a hammer and chisel—the result is not finished, but it begins to take shape.
"The model then optimizes the render to sharpen the details, just as a sculptor painstakingly hews stone to mimic the texture of fabric or a human figure.
The final result is a 3D object or large-scale scene that can be used in virtual reality applications, digital twins or robotics development," NVIDIA explains.
According to NVIDIA, previous AI models have struggled to get key details right for this sort of thing. Repetitive textures patterns, strong color variations, and other details have proven challenging for AI tools. That's not necessarily the case for Neuralangelo. In the video embedded above, you can see how NVIDIA's open-source tool recreates challenging objects such as Michelangelo's David, a flatbed truck, and the interior of a ballroom filled with chairs, tables, and other objects.
