20.7 NeuGPU: A 18.5mJ/Iter Neural-Graphics Processing Unit for Instant-Modeling and Real-Time Rendering with Segmented-Hashing Architecture

With the rise of the metaverse, there’s a growing demand for 3D modeling and rendering technologies that can bring real-world objects/scenes into the augmented/virtual world on mobile devices. Recently, 3D modeling and rendering using neural radiance field (NeRF) [1] are emerging as NeRF uses only 2D images to train a DNN and create a realistic 3D model without a user’s manual design or high-cost 3D scanner. Previous NeRF hardware [2] supported only high-fps 3D rendering of unseen views without the capability of instant modeling, because the vanilla NeRF model [1] takes 1-2 days to complete 3D modeling on a single NVIDIA V100 GPU. A recent NeRF model [3] enables relatively quick modeling by adopting hash embedding as shown in Fig. 20.7.1. However, it still spends >10 minutes for modeling and shows <1fps rendering speed while consuming >10 W power on a resource-constrained edge GPU [4] due to 3 main hardware challenges. First, the Hash Table (HT) has a huge number of parameters (>23 MB for 16 HT levels), but a typical L2 cache in edge devices (512KB on Jetson TX2) is not enough to store even a single level of HT. Moreover, irregular memory accesses associated with hash embedding increase the cache miss rate, and induce large off-chip memory access, consuming 90.0% of system energy. Second, on-chip memory access for trilinear interpolation of hash encoding suffers from throughput degradation caused by bank conflict. Although heavily-banked SRAM (32-banks/level) can mitigate the conflict, it consumes >80% of the interpolation power with area overhead. Third, unlike the vanilla NeRF acceleration, the low-sparsity characteristics (0-50%) of the hash-based NeRF [3] model cause low throughput and energy-efficiency for multi-layer perceptron (MLP) acceleration.