A Real-time Method for Inserting Virtual Objects into Neural Radiance Fields

We present the first real-time method for inserting a rigid virtual object into a neural radiance field (NeRF), which produces realistic lighting and shadowing effects, as well as allows interactive manipulation of the object. By exploiting the rich information about lighting and geometry in a NeRF, our method overcomes several challenges of object insertion in augmented reality. For lighting estimation, we produce accurate and robust incident lighting that combines the 3D spatially-varying lighting from NeRF and an environment lighting to account for sources not covered by the NeRF. For occlusion, we blend the rendered virtual object with the background scene using an opacity map integrated from the NeRF. For shadows, with a precomputed field of spherical signed distance fields, we query the visibility term for any point around the virtual object, and cast soft, detailed shadows onto 3D surfaces. Compared with state-of-the-art techniques, our approach can insert virtual objects into scenes with superior fidelity, and has great potential to be further applied to augmented reality systems.