Neural Rendering Acceleration With Deferred Neural Decoding and Voxel-Centric Data Flow

Neural radiance field has become a fundamental rendering technique across diverse applications such as augmented/virtual reality and autonomous driving. It achieves exceptional rendering quality and reduces model construction cost mainly by introducing a novel neural representation, Instant Neural Graphics Primitives (Instant-NGP). Despite its superiority, Instant-NGP poses severe problems of intensive computation, memory inefficiency and pipeline inefficiency, owing to numerous neural network queries, irregular memory access and intricate sampling procedure. To address these issues, this article proposes NeRA, an algorithm-architecture co-optimization framework that facilitates the efficient neural rendering of Instant-NGP. For intensive computation, we reconstruct the rendering flow and propose a deferred neural decoding algorithm to aggregate the network queries, which reduces the computational workload by 85.6% and only incurs <0.5 % loss in rendering quality. For memory inefficiency, a base-interpolation algorithm is proposed to condense the scattered memory access and improves the equivalent bandwidth of on-chip memory by 2.38×. Furthermore, a voxel-centric data flow is proposed to fully reuse the cached data and save 88.7% of the external memory access. For pipeline inefficiency, a highly-pipelined hardware architecture with decoupled spatial skipping and interleaved sampling is constructed to eliminate the bubbles and invalid samples in the pipeline, which boosts the overall throughput by 2.41×. Extensively evaluated on representative benchmarks, NeRA attains 1.2 2.9× in rendering throughput, 1.7 36.5× in energy-efficiency and 3.6 8.3× in area-efficiency, compared to the state-of-the-art related architectures.