As one might expect, creating a virtual 3D world is a laborious process that requires not only time and effort but also a high level of skill. Look at any video game, and the graphics will have required thousands and thousands of hours of work on the part of developers specialized in 3D rendering, lighting, textures and more.
That level of work to create virtual environments may not be necessary for much longer, however, as graphics processing unit company Nvidia has developed a powerful AI algorithm capable of generating photorealistic details for a virtual world. The AI tool could not only impact the video game development industry, but could also have important applications in generating environments for virtual reality as well as for training self-driving cars and robots.
Bryan Catanzaro, the vice president of applied deep learning at Nvidia, explained: “We can create new sketches that have never been seen before and render those. We’re actually teaching the model how to draw based on real video.”
If the AI tool sounds impressive, that’s because it is. Nvidia’s team has utilized a conventional machine-learning approach to identify various objects appearing in a video scene, such as cars, trees, homes, etc. From there, it has leveraged a more sophisticated generative adversarial network (GAN) to train a computer to generate photorealistic 3D imagery to fill the outlines of objects.
In other words, users can input the outline of a scene (indicating where different objects should be), and the algorithm will fill in the outlines with realistic details. At this stage, the graphics generated by the algorithm are not perfect—as some objects can appear lightly warped, almost dreamlike—but the output is already highly advanced.
“Classical computer graphics render by building up the way light interacts with objects,” added Catanzaro. “We wondered what we could do with artificial intelligence to change the rendering process.”
The innovative approach was presented at NeurIPS, an AI conference hosted in Montreal this week and has gained significant attention already, largely from the gaming sector, which could drastically improve (in terms of ease) game design.
“NVIDIA has been inventing new ways to generate interactive graphics for 25 years, and this is the first time we can do so with a neural network,” said Catanzaro. “Neural networks — specifically generative models — will change how graphics are created. This will enable developers to create new scenes at a fraction of the traditional cost.”
Located in Santa Clara, California, Nvidia is the latest company to unveil a GAN-based algorithm. Invented by Google, GANs have only been around for a few years but are already demonstrating their ability to produce realistic imagery and sounds. You might recall a viral video of Trump from May 2018 calling on the Belgian people to withdraw from the Paris climate agreement. The video—though clearly fake—was created using GANs and demonstrated the potentially dangerous ramifications of the technology on creating realistic and (actually fake) fake news.
Still, from a technology point of view, Nvidia’s algorithm is mighty impressive and we’re eager to see what its first real world (virtual world?) applications are.