Nandita Vijaykumar and Igor Gilitschenski are co-recipients of a 2022 Sony Focused Research Award.
How can reconstructing a 3D representation of a scene from 2D images be performed efficiently on smaller, low-power battery-operated devices?
A neural radiance field (NeRF) is a transformative deep learning technique used in computer graphics and computer vision, particularly for the representation and rendering of 3D scenes. However, the significant amount of computation it requires is both time consuming and expensive to support.
Nandita Vijaykumar and Igor Gilitschenski, both assistant professors in the Department of Computer Science, have jointly received a 2022 Sony Focused Research Award to explore this challenge. The award provides funding for “pioneering research that could drive new technologies, industries and the future.”
“It is great to start a new collaboration with a major company on an exciting and important project, and we hope we can turn this into a bigger longer-term collaboration with U of T,” says Vijaykumar, who is also an assistant professor in the Department of Computer and Mathematical Sciences at the University of Toronto Scarborough.
Their project, “Fast and Efficient Online 3D Reconstruction on Resource-Constrained Devices using Implicit Neural Representations” seeks to develop a framework to perform efficient and versatile 3D reconstruction on resource-constrained mobile devices, using implicit representations and neural rendering.
To address the high computational demands of using NeRFs, the researchers are taking an interdisciplinary approach that leverages hardware and software co-design and draws on new innovations in computer vision, robotics and machine learning optimization.
“We are excited about speeding up neural rendering. This technology holds huge potential for robotics and augmented reality,” says Gilitschenski, who is also an assistant professor in the Department of Mathematical and Computational Sciences at the University of Toronto Mississauga.
Their envisioned framework would make NeRFs feasible for important tasks that are currently infeasible due to the high computational demands. This includes accurate 3D reconstruction of small and large-scale scenes using handheld and small devices.
Additionally, the framework would facilitate the use of machine learning-based rendering for virtual reality, rapid scene interpretation using neural representations in wearable devices for augmented reality, for remote sensing and diagnosis with medical devices, and simultaneous localization and mapping (SLAM) in small autonomous robots.
The use of NeRFs has wide-ranging applications, from immersive virtual and augmented reality experiences, aiding in medical imaging to preserving cultural heritage, 3D content creation and reshaping how we visualize and interact with the digital world.
According to the researchers, tackling this technology’s challenge has the potential to significantly enhance real-time rendering capabilities, catalyze innovation and productivity in its application, and enable its use in smaller devices such as mobile phones, medical devices and robots.