An Artificial-Intelligence-based SLAM Processor with Scene-adaptive Sampling and Hybrid NeRF Model Training Acceleration

This paper presents an energy-efficient artificial intelligence (AI) based SLAM processor to provide camera pose together with red, green, blue, and depth (RGB-D) reconstruction of the environment near the user. It acquires such information in real-time through the tracking and mapping process that performs deep neural network (DNN) training on neural radiance fields (NeRF) model by receiving RGB-D camera input as the ground truth, without relying on any prior data in indoor and unknown user environments. However, the realization of AI-based SLAM on performance- and battery-constrained devices is extremely challenging due to the massive computation amount and heterogeneous characteristics at different DNN training stages and layers. To satisfy the hardware requirements of SLAM in mobile devices, this article proposed three key features. A Scene-adaptive Sampling (SAS) algorithm is proposed and implemented in the sampling core to detect and eliminate energy consumption caused by redundant workloads at both the tracking and mapping procedures. After preprocessing at the sampling core, training the hybrid NeRF model which consists of 3D embedding and sparse mixture-of-experts (SMoE) layers is accelerated for high throughput. 3D embedding unit (3D-EU) provides the data-path that is shareable among feed-forward (FF), back-propagation (BP), and weight-gradient (WG) stages of the 3D embedding layer. Moreover, SMoE acceleration architecture schedules the execution order of SMoE layer to maximize the reuse amount of expert parameters. Finally, the proposed real-time SLAM processor achieves 0.14 μJ/point energy consumption, which is 2.86× smaller than the state-of-the-art.