Bringing NeRFs to the Latent Space: Inverse Graphics Autoencoder

April 2, 2025

A novel IG-AE that creates a 3D-aware latent space compatible with Neural Radiance Fields

This Research Card introduces a novel Inverse Graphics Autoencoder (IG-AE) that creates a 3D-aware latent space compatible with Neural Radiance Fields (NeRFs), enabling efficient 3D object reconstruction from 2D images and facilitating high-quality 3D ads.

Title: Bringing NeRFs to the Latent Space: Inverse Graphics Autoencoder
Project page: https://ig-ae.github.io
Short Title: Inverse Graphics Autoencoder
Authors: Antoine Schnepf*, Karim Kassab*, Jean-Yves Franceschi, Laurent Caraffa, Flavian Vasile, Jeremie Mary, Andrew Comport, Valérie Gouet-Brunet
Team: R&D — CAIL — AIR — AI Foundations — Commerce Foundation Models
Revue: International Conference on Learning Representations (ICLR) 2025
Status: Accepted
Category: Deep Learning, Computer vision, 3D

Why did we work on this topic, the problem we want to solve?

Recent advancements in deep learning and generative AI have enabled the reconstruction of 3D objects directly from images, a process that traditionally relied on complex 3D acquisition techniques. Techniques like Neural Radiance Fields (NeRFs) now allow for the extraction of 3D representations from 2D images, but this extraction is often costly due to high-resolution rendering. By applying this process in a lower-resolution latent space, we can reduce training and rendering complexity, opening up latent-based applications currently limited to 2D images. However, existing latent spaces typically lack an underlying 3D geometry, resulting in poor-quality latent NeRFs.

What did we find? What did we achieve?

Building on the previous discourse, we propose a new Inverse Graphics Autoencoder (IG-AE) that embeds a latent space that is compatible with NeRF learning, which we call “3D-aware” latent space.

This enables accelerated NeRF training in the latent space,
It reduces quality losses with respect to a standard latent space.

Additionally, we propose an open-source Nerfstudio extension in which various latent NeRFs can be trained.

How did we proceed?

We use a large set of synthetic 3D data as supervision for our (2D) latent space. By doing so, we obtain our 3D-aware latent space that is compatible with NeRFs. We also ensure that this property generalizes outside of the training data.

What is the originality here?

Learning scenes in a 3D-aware latent space is a novel and promising approach that has largely remained unexplored. We are the first to propose a 3D-aware latent space, and a general approach for learning latent NeRFs.

Illustration

Below, we illustrate how an object can be learned in 3D from its captured images using our proposed latent NeRF training. In the first row, the object is learned in a traditional latent space. In the second row, it is learned in our 3D-aware latent space, with superior quality.
The first column showcases the object in the latent space, while the others showcase its decoding to the RGB space.