MetaVRain: A Mobile Neural 3-D Rendering Processor With Bundle-Frame-Familiarity-Based NeRF Acceleration and Hybrid DNN Computing

This article presents MetaVRain, a low-power neural 3-D rendering processor for metaverse realization on mobile devices. The MetaVRain mainly focused on solving a high operational intensity problem that appeared during the neural radiance fields (NeRFs)-based rendering. It imitates brain-inspired visual perception processes and constructs a new NeRF acceleration architecture, bundle-frame-familiarity (BuFF). The built-in visual perception core (VPC) realizes BuFF architecture by accelerating three visual perception stages: 1) spatial attention (SA); 2) temporal familiarity (TF); and 3) top-down attention (TDA). Both TF unit (TFU) and SA unit (SAU) included in VPC help the processor not to suffer from external memory bandwidth. Moreover, unlike conventional GPU solutions, they remove useless operations that do not cause PSNR drop using an out-of-order task allocator. After removing useless operations, the remaining deep neural network (DNN) inference (INF) tasks are accelerated by a hybrid neural engine (HNE) which utilizes two different neural engines (NEs) 1-D and 2-D NEs. It uses 1-D NE for input activation (IA) sparsity exploitation and 2-D NE to maximize the data reusability. Thanks to the centrifugal sampling (CS)-based output sparsity prediction, it can dynamically allocate tasks to 1-D or 2-D NE. In addition, it accelerates the positional encoding part by adopting periodic polynomial-based sinusoidal function approximation. The MetaVRain suggests a modulo-based positional encoding unit (Mod-PEU) to realize sinusoidal function generator circuits with low-power consumption and low area occupation. The MetaVRain is fabricated in a 28-nm process and examined with both the public and custom datasets. It finally achieves a maximum of 118 frames/s while consuming 99.95% lower power compared with modern GPUs.