***** Princeton Robotics Seminar: Mac Schwager, TODAY at 3pm ***** Time: 3pm EST, Friday Apr 22, 2022 Location: Zoom [ https://princeton.zoom.us/my/robotics | link ] Speaker: [ https://web.stanford.edu/~schwager/ | Mac Schwager ] , Stanford AeroAstro Title: Reimagining Robot Autonomy with Neural Environment Representations Abstract: New developments in computer vision and deep learning have led to the rise of neural environment representations: 3D maps that are stored as deep networks that spatially register occupancy, color, texture, and other physical properties. These environment models can generate photo-realistic synthetic images from unseen view points, and can store 3D information in exquisite detail. In this talk, I investigate the question: How can robots use neural environment representations for perception, motion planning, manipulation, and simulation? I will show recent work from my lab in which we build a robot navigation pipeline using a Neural Radiance Field (NeRF) map of an environment. We develop a trajectory optimization algorithm that interfaces with the NeRF model to find dynamically feasible, collision-free trajectories for a robot moving through a NeRF world. We also develop an optimization-based state estimator that uses the NeRF model to give full dynamic state estimates for a robot from only on board images. I will also discuss preliminary results using NeRF models for grasp planning, and for tracking the poses of multiple 3D objects in a scene. Finally, I will discuss a new differentiable robot physics simulator called Dojo that can use NeRFs as a geometry description for objects, leading to physically interpretable motion prediction from NeRF models. I will conclude with future opportunities and challenges in integrating neural environment representations into the robot autonomy stack. Bio: Mac Schwager is an Associate Professor of Aeronautics and Astronautics at Stanford University. He directs the Multi-robot Systems Lab (MSL) where he studies distributed algorithms for control, perception, and learning in groups of robots and autonomous systems. He is interested in a range of applications including cooperative surveillance with teams of UAVs, autonomous driving in traffic, cooperative robotic manipulation, distributed SLAM, distributed trajectory planning, and autonomous drone and car racing. He obtained his BS degree from Stanford, and his MS and PhD degrees from MIT. He was a postdoctoral researcher at the University of Pennsylvania and at MIT. Prior to joining Stanford, he was an assistant professor at Boston University from 2012 to 2015. He received the NSF CAREER award in 2014, the DARPA YFA in 2018, and has received numerous best paper awards in conferences and journals including the IEEE Transactions on Robotics best paper award in 2016, the Best Paper Award in Robot Manipulation in ICRA 2018, and the Best Paper Award in Multi-Robot Systems in ICRA 2020.