Abstract:
Neuromorphic cameras are emerging imaging technology that has advantages over conventional imaging sensors in several aspects including dynamic range, sensing latency, an...Show MoreMetadata
Abstract:
Neuromorphic cameras are emerging imaging technology that has advantages over conventional imaging sensors in several aspects including dynamic range, sensing latency, and power consumption. However, the signal-to-noise level and the spatial resolution still fall behind the state of conventional imaging sensors. In this article, we address the denoising and super-resolution problem for modern neuromorphic cameras. We employ 3D U-Net as the backbone neural architecture for such a task. The networks are trained and tested on two types of neuromorphic cameras: a dynamic vision sensor and a spike camera. Their pixels generate signals asynchronously, the former is based on perceived light changes and the latter is based on accumulated light intensity. To collect the datasets for training such networks, we design a display-camera system to record high frame-rate videos at multiple resolutions, providing supervision for denoising and super-resolution. The networks are trained in a noise-to-noise fashion, where the two ends of the network are unfiltered noisy data. The output of the networks has been tested for downstream applications including event-based visual object tracking and image reconstruction. Experimental results demonstrate the effectiveness of improving the quality of neuromorphic events and spikes, and the corresponding improvement to downstream applications with state-of-the-art performance.
Published in: IEEE Transactions on Pattern Analysis and Machine Intelligence ( Volume: 45, Issue: 12, December 2023)
Funding Agency:
National Key Laboratory for Multimedia Information Processing and National Engineering Research Center of Visual Technology, School of Computer Science, Peking University, Beijing, China
Peiqi Duan received the PhD degree from Peking University, in 2023. He is currently a Boya postdoctoral fellow with the School of Computer Science, Peking University. His research interests span event-based imaging and vision, single-image super-resolution, and HDR image reconstruction. He has served as a reviewer/program committee member of the International Journal of Computer Vision, IEEE Transactions on Circuits and S...Show More
Peiqi Duan received the PhD degree from Peking University, in 2023. He is currently a Boya postdoctoral fellow with the School of Computer Science, Peking University. His research interests span event-based imaging and vision, single-image super-resolution, and HDR image reconstruction. He has served as a reviewer/program committee member of the International Journal of Computer Vision, IEEE Transactions on Circuits and S...View more
National Key Laboratory for Multimedia Information Processing and National Engineering Research Center of Visual Technology, School of Computer Science, Peking University, Beijing, China
Yi Ma received the BS degree from Peking University, in 2021. He is currently working toward the graduate degree with Peking University. His research interests include centered around event-based vision signal processing.
Yi Ma received the BS degree from Peking University, in 2021. He is currently working toward the graduate degree with Peking University. His research interests include centered around event-based vision signal processing.View more
National Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University, Beijing, China
Xinyu Zhou received the BS degree from Peking University, in 2022. He is currently working toward the PhD degree with the School of Intelligence Science and Technology, Peking University. His research interests include computational photography and event-based vision.
Xinyu Zhou received the BS degree from Peking University, in 2022. He is currently working toward the PhD degree with the School of Intelligence Science and Technology, Peking University. His research interests include computational photography and event-based vision.View more
National Key Laboratory for Multimedia Information Processing and National Engineering Research Center of Visual Technology, School of Computer Science, Peking University, Beijing, China
Xinyu Shi is currently working toward the PhD degree with the School of Electronic Engineering and Computer Science, Peking University. His research interests include event-based vision signal processing and spiking neuron network.
Xinyu Shi is currently working toward the PhD degree with the School of Electronic Engineering and Computer Science, Peking University. His research interests include event-based vision signal processing and spiking neuron network.View more
Northwestern University, Evanston, IL, USA
Zihao W. Wang received the diploma (honors) degree in pptics from Chu Ko-Chen Honors College, Zhejiang University, in 2015, and the PhD degree from Northwestern University, in 2020. He is a computer vision engineer with Apple Inc. His research interests include computer vision and machine learning. He has served as a reviewer/program committee member in CVPR, ECCV, NeurIPS, ICLR, AAAI, etc.
Zihao W. Wang received the diploma (honors) degree in pptics from Chu Ko-Chen Honors College, Zhejiang University, in 2015, and the PhD degree from Northwestern University, in 2020. He is a computer vision engineer with Apple Inc. His research interests include computer vision and machine learning. He has served as a reviewer/program committee member in CVPR, ECCV, NeurIPS, ICLR, AAAI, etc.View more
National Key Laboratory for Multimedia Information Processing and National Engineering Research Center of Visual Technology, School of Computer Science, Peking University, Beijing, China
Tiejun Huang (Senior Member, IEEE) received the BSc and MSc degrees in computer science from the Wuhan University of Technology, China, in 1992 and 1995, respectively, and the PhD degree in pattern recognition and image analysis from the Huazhong (Central China) University of Science and Technology, in 1998. He is currently a professor with the School of Computer Science, Peking University, and the director of the Beijing...Show More
Tiejun Huang (Senior Member, IEEE) received the BSc and MSc degrees in computer science from the Wuhan University of Technology, China, in 1992 and 1995, respectively, and the PhD degree in pattern recognition and image analysis from the Huazhong (Central China) University of Science and Technology, in 1998. He is currently a professor with the School of Computer Science, Peking University, and the director of the Beijing...View more
National Key Laboratory for Multimedia Information Processing and National Engineering Research Center of Visual Technology, School of Computer Science, Peking University, Beijing, China
Boxin Shi (Senior Member, IEEE) received the BE degree from the Beijing University of Posts and Telecommunications, in 2007, the ME degree from Peking University, in 2010, and the PhD degree from the University of Tokyo, in 2013. He is currently a Boya Young fellow assistant professor and research professor with Peking University, where he leads the Camera Intelligence Lab. Before joining PKU, he did research with MIT Med...Show More
Boxin Shi (Senior Member, IEEE) received the BE degree from the Beijing University of Posts and Telecommunications, in 2007, the ME degree from Peking University, in 2010, and the PhD degree from the University of Tokyo, in 2013. He is currently a Boya Young fellow assistant professor and research professor with Peking University, where he leads the Camera Intelligence Lab. Before joining PKU, he did research with MIT Med...View more
Contents National Key Laboratory for Multimedia Information Processing and National Engineering Research Center of Visual Technology, School of Computer Science, Peking University, Beijing, China
Peiqi Duan received the PhD degree from Peking University, in 2023. He is currently a Boya postdoctoral fellow with the School of Computer Science, Peking University. His research interests span event-based imaging and vision, single-image super-resolution, and HDR image reconstruction. He has served as a reviewer/program committee member of the International Journal of Computer Vision, IEEE Transactions on Circuits and Systems for Video Technology, CVPR, ICCV, ECCV, NeurIPS, etc.
Peiqi Duan received the PhD degree from Peking University, in 2023. He is currently a Boya postdoctoral fellow with the School of Computer Science, Peking University. His research interests span event-based imaging and vision, single-image super-resolution, and HDR image reconstruction. He has served as a reviewer/program committee member of the International Journal of Computer Vision, IEEE Transactions on Circuits and Systems for Video Technology, CVPR, ICCV, ECCV, NeurIPS, etc.View more
National Key Laboratory for Multimedia Information Processing and National Engineering Research Center of Visual Technology, School of Computer Science, Peking University, Beijing, China
Yi Ma received the BS degree from Peking University, in 2021. He is currently working toward the graduate degree with Peking University. His research interests include centered around event-based vision signal processing.
Yi Ma received the BS degree from Peking University, in 2021. He is currently working toward the graduate degree with Peking University. His research interests include centered around event-based vision signal processing.View more
National Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University, Beijing, China
Xinyu Zhou received the BS degree from Peking University, in 2022. He is currently working toward the PhD degree with the School of Intelligence Science and Technology, Peking University. His research interests include computational photography and event-based vision.
Xinyu Zhou received the BS degree from Peking University, in 2022. He is currently working toward the PhD degree with the School of Intelligence Science and Technology, Peking University. His research interests include computational photography and event-based vision.View more
National Key Laboratory for Multimedia Information Processing and National Engineering Research Center of Visual Technology, School of Computer Science, Peking University, Beijing, China
Xinyu Shi is currently working toward the PhD degree with the School of Electronic Engineering and Computer Science, Peking University. His research interests include event-based vision signal processing and spiking neuron network.
Xinyu Shi is currently working toward the PhD degree with the School of Electronic Engineering and Computer Science, Peking University. His research interests include event-based vision signal processing and spiking neuron network.View more
Northwestern University, Evanston, IL, USA
Zihao W. Wang received the diploma (honors) degree in pptics from Chu Ko-Chen Honors College, Zhejiang University, in 2015, and the PhD degree from Northwestern University, in 2020. He is a computer vision engineer with Apple Inc. His research interests include computer vision and machine learning. He has served as a reviewer/program committee member in CVPR, ECCV, NeurIPS, ICLR, AAAI, etc.
Zihao W. Wang received the diploma (honors) degree in pptics from Chu Ko-Chen Honors College, Zhejiang University, in 2015, and the PhD degree from Northwestern University, in 2020. He is a computer vision engineer with Apple Inc. His research interests include computer vision and machine learning. He has served as a reviewer/program committee member in CVPR, ECCV, NeurIPS, ICLR, AAAI, etc.View more
National Key Laboratory for Multimedia Information Processing and National Engineering Research Center of Visual Technology, School of Computer Science, Peking University, Beijing, China
Tiejun Huang (Senior Member, IEEE) received the BSc and MSc degrees in computer science from the Wuhan University of Technology, China, in 1992 and 1995, respectively, and the PhD degree in pattern recognition and image analysis from the Huazhong (Central China) University of Science and Technology, in 1998. He is currently a professor with the School of Computer Science, Peking University, and the director of the Beijing Academy for Artificial Intelligence. His research areas include visual information processing and neuromorphic computing. He is a fellow of CAAI, CCF, CSIG, and vice chair of the China National General Group on AI Standardization. He published more than 300 peer-reviewed papers on leading journals and conferences, and also co-editor of four ISO/IEC standards, five National standards and four IEEE standards. He holds more than 100 granted patents. He received National Award for Science and Technology of China (Tier-2) for three times (2010, 2012, 2017).
Tiejun Huang (Senior Member, IEEE) received the BSc and MSc degrees in computer science from the Wuhan University of Technology, China, in 1992 and 1995, respectively, and the PhD degree in pattern recognition and image analysis from the Huazhong (Central China) University of Science and Technology, in 1998. He is currently a professor with the School of Computer Science, Peking University, and the director of the Beijing Academy for Artificial Intelligence. His research areas include visual information processing and neuromorphic computing. He is a fellow of CAAI, CCF, CSIG, and vice chair of the China National General Group on AI Standardization. He published more than 300 peer-reviewed papers on leading journals and conferences, and also co-editor of four ISO/IEC standards, five National standards and four IEEE standards. He holds more than 100 granted patents. He received National Award for Science and Technology of China (Tier-2) for three times (2010, 2012, 2017).View more
National Key Laboratory for Multimedia Information Processing and National Engineering Research Center of Visual Technology, School of Computer Science, Peking University, Beijing, China
Boxin Shi (Senior Member, IEEE) received the BE degree from the Beijing University of Posts and Telecommunications, in 2007, the ME degree from Peking University, in 2010, and the PhD degree from the University of Tokyo, in 2013. He is currently a Boya Young fellow assistant professor and research professor with Peking University, where he leads the Camera Intelligence Lab. Before joining PKU, he did research with MIT Media Lab, Singapore University of Technology and Design, Nanyang Technological University, National Institute of Advanced Industrial Science and Technology, from 2013 to 2017. His papers were awarded as Best Paper Runner-Up at ICCP 2015 and selected as Best Papers from ICCV 2015 for IJCV Special Issue. He has served as an associate editor of the IEEE Transactions on Pattern Analysis and Machine Intelligence/International Journal of Computer Vision and an area chair of CVPR/ICCV.
Boxin Shi (Senior Member, IEEE) received the BE degree from the Beijing University of Posts and Telecommunications, in 2007, the ME degree from Peking University, in 2010, and the PhD degree from the University of Tokyo, in 2013. He is currently a Boya Young fellow assistant professor and research professor with Peking University, where he leads the Camera Intelligence Lab. Before joining PKU, he did research with MIT Media Lab, Singapore University of Technology and Design, Nanyang Technological University, National Institute of Advanced Industrial Science and Technology, from 2013 to 2017. His papers were awarded as Best Paper Runner-Up at ICCP 2015 and selected as Best Papers from ICCV 2015 for IJCV Special Issue. He has served as an associate editor of the IEEE Transactions on Pattern Analysis and Machine Intelligence/International Journal of Computer Vision and an area chair of CVPR/ICCV.View more
