A 33.6 FPS Embedding based Real-time Neural Rendering Accelerator with Switchable Computation Skipping Architecture on Edge Device

A neural radiance field (NeRF) algorithm which utilizes a deep neural network (DNN) is rising as a new realistic 3D rendering technology. Recently, explicit NeRF [1] that utilizes interpolation on 3D embedding grids benefits 22% more compact memory footprint compared with implicit NeRF which is fully based on multi-layer perceptron (MLP) operations. Fig. 1 shows embedding-based explicit NeRF. It consists of 3 computing stages: 1) sampling, 2) density computation, and 3) color computation. The sampling stage casts rays from each image pixel and determines whether the sample is valid or not. The density computation stage performs interpolation between the two nearest embedding values from pre-decomposed X, Y, and Z density embedding vectors and sums up the interpolated values along ranks. If the calculated density is above the threshold, the color computation is performed. The color computation stage computes interpolation and multiplying basis matrix.