The most successful methods for image understanding tasks rely on feed-forward neural networks. But despite their empirical accuracy and efficiency, prediction using deep neural networks is not without fundamental disadvantages. Large parts of the networks are often difficult to analyze in failure or borderline cases. This is true in particular when attempting to perceive 3D information from 2D images. Formulating perceptive tasks as a probabilistic generation of world representations has been explored for years under the paradigm of analysis by synthesis, where models of the world help interpret observations. In this exam I present 3D perception tasks recast as an Inverse Neural Rendering problem, reconstructing complex 3D environments to reason about the underlying world captured by images and videos. Since this is particularly important for safety-critical robotics applications like autonomous driving, this will be the focus of the presented work with applications on object detection and 3D multi-object tracking.

Recent progress in Neural Rendering and generative models enable the synthesis of plausible representations from input images. But synthesizing dynamic, large-scale outdoor scenes for autonomous driving comes with its own set of challenges that a lot of approaches did not handle. Neural Scene Graphs and Point Light Fields present two solutions to those aspects. Through the integration of generative priors this can be transferred from reconstruction of a single scene to similar scenes, exemplified with work that uses diffusion probabilistic models to generate intermediate layout representation suited to detect objects in a scene and generative object models to efficiently reconstruct objects in a scene and allow more interpretable object tracking utilizing the underlying shape and texture.

https://docs.google.com/document/d/1e4qtySO8em4Wtjc_VwMfCcsRLqFgJqAAAhl2PLVrE8M/edit?usp=sharing