Multibit Attention Fusion for Gaze Estimation Using 12-Bit RAW Data From CMOS Sensors

In recent years, deep learning-based gaze estimation techniques using eye images have made significant progress. However, balancing prediction accuracy and computational complexity remains a challenge. In this article, we propose using the RAW data from a CMOS image sensor (CIS) as input, thereby avoiding the high computational demands of the traditional image signal processor (ISP) (which converts RAW to RGB). We designed a special measuring scheme to collect and annotate a 12-bit RAW dataset. Additionally, we introduce the Multibit Attention Fusion Gaze Estimation Network (MAFN) to accurately perform gaze estimation using 12-bit RAW data. Our MAFN effectively extracts features of varying granularities through attention fusion techniques across different bit depths. We also employ axial sensitivity spatial attention, which is more sensitive to the horizontal and vertical positions of the pupil in gaze angle prediction. Furthermore, we propose bit rolling channelization and neural network-friendly quantization (NNFQ) for lossless input of 12-bit data and low-bit data generation, respectively. Experimental results validate the effectiveness of MAFN and demonstrate the superiority of our method, showing that it can accurately measure gaze direction using 12-bit RAW data.