Abstract:
Light field (LF) photography is an emerging paradigm for capturing more immersive representations of the real world. However, arising from the inherent tradeoff between t...Show MoreMetadata
Abstract:
Light field (LF) photography is an emerging paradigm for capturing more immersive representations of the real world. However, arising from the inherent tradeoff between the angular and spatial dimensions, the spatial resolution of LF images captured by commercial micro-lens-based LF cameras is significantly constrained. In this paper, we propose effective and efficient end-to-end convolutional neural network models for spatially super-resolving LF images. Specifically, the proposed models have an hourglass shape, which allows feature extraction to be performed at the low-resolution level to save both the computational and memory costs. To fully make use of the 4D structure information of LF data in both the spatial and angular domains, we propose to use 4D convolution to characterize the relationship among pixels. Moreover, as an approximation of 4D convolution, we also propose to use spatial-angular separable (SAS) convolutions for more computationally and memory-efficient extraction of spatial-angular joint features. Extensive experimental results on 57 test LF images with various challenging natural scenes show significant advantages from the proposed models over the state-of-the-art methods. That is, an average PSNR gain of more than 3.0 dB and better visual quality are achieved, and our methods preserve the LF structure of the super-resolved LF images better, which is highly desirable for subsequent applications. In addition, the SAS convolution-based model can achieve three times speed up with only negligible reconstruction quality decrease when compared with the 4D convolution-based one. The source code of our method is available online.
Published in: IEEE Transactions on Image Processing ( Volume: 28, Issue: 5, May 2019)
Funding Agency:
School of Information Technologies, The University of Sydney, Sydney, NSW, Australia
Henry Wing Fung Yeung received the B.A. degree in economics from the University of Cambridge in 2015 and the M.Phil. degree in engineering and IT from the The University of Sydney in 2017, where he is currently pursuing the Ph.D. degree in engineering and IT. He has authored or co-authored multiple international journal and conference papers, including the European Conference on Computer Vision. His research interests inc...Show More
Henry Wing Fung Yeung received the B.A. degree in economics from the University of Cambridge in 2015 and the M.Phil. degree in engineering and IT from the The University of Sydney in 2017, where he is currently pursuing the Ph.D. degree in engineering and IT. He has authored or co-authored multiple international journal and conference papers, including the European Conference on Computer Vision. His research interests inc...View more
City University of Hong Kong Shenzhen Research Institute, Shenzhen, China
Junhui Hou (S’13–M’16) received the B.Eng. degree in information engineering (Talented Students Program) from the South China University of Technology, Guangzhou, China, in 2009, the M.Eng. degree in signal and information processing from Northwestern Polytechnical University, Xian, China, in 2012, and the Ph.D. degree from the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, i...Show More
Junhui Hou (S’13–M’16) received the B.Eng. degree in information engineering (Talented Students Program) from the South China University of Technology, Guangzhou, China, in 2009, the M.Eng. degree in signal and information processing from Northwestern Polytechnical University, Xian, China, in 2012, and the Ph.D. degree from the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, i...View more
Institute of Advanced Technology, University of Science and Technology of China, Hefei, China
Xiaoming Chen received the B.Sc. degree from the Royal Melbourne Institute of Technology, Australia, and the Ph.D. degree from The University of Sydney, Australia, in 2004 and 2009, respectively. From 2010 to 2014, he was with the National University of Singapore, CSIRO Australia, and IBM. He is currently with the School of Information Technologies, The University of Sydney. He is also a Researcher with the Institute of A...Show More
Xiaoming Chen received the B.Sc. degree from the Royal Melbourne Institute of Technology, Australia, and the Ph.D. degree from The University of Sydney, Australia, in 2004 and 2009, respectively. From 2010 to 2014, he was with the National University of Singapore, CSIRO Australia, and IBM. He is currently with the School of Information Technologies, The University of Sydney. He is also a Researcher with the Institute of A...View more
School of Electrical and Electronics Engineering, Nanyang Technological University, Singapore
Jie Chen (S’13–M’16) received the B.S. and M.Eng. degrees from the School of Optical and Electronic Information, Huazhong University of Science and Technology, China, in 2008 and 2011, respectively, and the Ph.D. degree from the School of Electrical and Electronic Engineering, Nanyang Technological University (NTU), Singapore, in 2016. He is currently a Research Fellow with the ST Engineering-NTU Corporate Laboratory. His...Show More
Jie Chen (S’13–M’16) received the B.S. and M.Eng. degrees from the School of Optical and Electronic Information, Huazhong University of Science and Technology, China, in 2008 and 2011, respectively, and the Ph.D. degree from the School of Electrical and Electronic Engineering, Nanyang Technological University (NTU), Singapore, in 2016. He is currently a Research Fellow with the ST Engineering-NTU Corporate Laboratory. His...View more
School of Information Science and Technology, University of Science and Technology of China, Hefei, China
Zhibo Chen (M’01–SM’11) received the B.Sc. and Ph.D. degrees from the Department of Electrical Engineering, Tsinghua University, in 1998 and 2003, respectively. He was with SONY and Thomson from 2003 to 2012. He was a Principal Scientist and a Research Manager at the Thomson Research & Innovation Department. He is currently a Professor with the University of Science and Technology of China. He has more than 50 granted and...Show More
Zhibo Chen (M’01–SM’11) received the B.Sc. and Ph.D. degrees from the Department of Electrical Engineering, Tsinghua University, in 1998 and 2003, respectively. He was with SONY and Thomson from 2003 to 2012. He was a Principal Scientist and a Research Manager at the Thomson Research & Innovation Department. He is currently a Professor with the University of Science and Technology of China. He has more than 50 granted and...View more
School of Information Technologies, The University of Sydney, Sydney, NSW, Australia
Yuk Ying Chung received the B.Sc. degree in computing and information systems from the University of London in 1995 and the Ph.D. degree in computer engineering from the Queensland University of Technology, Australia, in 2000. From 1999 to 2001, she was a Lecturer at La Trobe University, Melbourne, Australia. Since 2001, she has been with the School of Information Technologies, The University of Sydney, Australia. Her res...Show More
Yuk Ying Chung received the B.Sc. degree in computing and information systems from the University of London in 1995 and the Ph.D. degree in computer engineering from the Queensland University of Technology, Australia, in 2000. From 1999 to 2001, she was a Lecturer at La Trobe University, Melbourne, Australia. Since 2001, she has been with the School of Information Technologies, The University of Sydney, Australia. Her res...View more
Contents School of Information Technologies, The University of Sydney, Sydney, NSW, Australia
Henry Wing Fung Yeung received the B.A. degree in economics from the University of Cambridge in 2015 and the M.Phil. degree in engineering and IT from the The University of Sydney in 2017, where he is currently pursuing the Ph.D. degree in engineering and IT. He has authored or co-authored multiple international journal and conference papers, including the European Conference on Computer Vision. His research interests include light field image processing and machine learning.
Henry Wing Fung Yeung received the B.A. degree in economics from the University of Cambridge in 2015 and the M.Phil. degree in engineering and IT from the The University of Sydney in 2017, where he is currently pursuing the Ph.D. degree in engineering and IT. He has authored or co-authored multiple international journal and conference papers, including the European Conference on Computer Vision. His research interests include light field image processing and machine learning.View more
City University of Hong Kong Shenzhen Research Institute, Shenzhen, China
Junhui Hou (S’13–M’16) received the B.Eng. degree in information engineering (Talented Students Program) from the South China University of Technology, Guangzhou, China, in 2009, the M.Eng. degree in signal and information processing from Northwestern Polytechnical University, Xian, China, in 2012, and the Ph.D. degree from the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, in 2016.
He has been an Assistant Professor with the Department of Computer Science, City University of Hong Kong, since 2017. His current research interests include multimedia signal processing, such as adaptive image/video representations and analysis (RGB/depth/light field/hyperspectral), static/dynamic 3D geometry representations and processing (mesh/point cloud/MoCap), and discriminative modeling for clustering/classification.
Dr. Hou was a recipient of the Prestigious Award from the Chinese Government for Outstanding Self-Financed Students Abroad, China Scholarship Council, in 2015, and the Early Career Award from the Hong Kong Research Grants Council in 2018. He currently serves as an Associate Editor for The Visual Computer and the Guest Editor for the Journal of Visual Communication and Image Representation.
Junhui Hou (S’13–M’16) received the B.Eng. degree in information engineering (Talented Students Program) from the South China University of Technology, Guangzhou, China, in 2009, the M.Eng. degree in signal and information processing from Northwestern Polytechnical University, Xian, China, in 2012, and the Ph.D. degree from the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, in 2016.
He has been an Assistant Professor with the Department of Computer Science, City University of Hong Kong, since 2017. His current research interests include multimedia signal processing, such as adaptive image/video representations and analysis (RGB/depth/light field/hyperspectral), static/dynamic 3D geometry representations and processing (mesh/point cloud/MoCap), and discriminative modeling for clustering/classification.
Dr. Hou was a recipient of the Prestigious Award from the Chinese Government for Outstanding Self-Financed Students Abroad, China Scholarship Council, in 2015, and the Early Career Award from the Hong Kong Research Grants Council in 2018. He currently serves as an Associate Editor for The Visual Computer and the Guest Editor for the Journal of Visual Communication and Image Representation.View more
Institute of Advanced Technology, University of Science and Technology of China, Hefei, China
Xiaoming Chen received the B.Sc. degree from the Royal Melbourne Institute of Technology, Australia, and the Ph.D. degree from The University of Sydney, Australia, in 2004 and 2009, respectively. From 2010 to 2014, he was with the National University of Singapore, CSIRO Australia, and IBM. He is currently with the School of Information Technologies, The University of Sydney. He is also a Researcher with the Institute of Advanced Technology, University of Science and Technology of China. His work has been published in ACM Multimedia, IEEE Virtual Reality, and IEEE T-CSVT. He has also invited more than 10 patents in his research areas. His research interests include immersive media computing and virtual reality.
Xiaoming Chen received the B.Sc. degree from the Royal Melbourne Institute of Technology, Australia, and the Ph.D. degree from The University of Sydney, Australia, in 2004 and 2009, respectively. From 2010 to 2014, he was with the National University of Singapore, CSIRO Australia, and IBM. He is currently with the School of Information Technologies, The University of Sydney. He is also a Researcher with the Institute of Advanced Technology, University of Science and Technology of China. His work has been published in ACM Multimedia, IEEE Virtual Reality, and IEEE T-CSVT. He has also invited more than 10 patents in his research areas. His research interests include immersive media computing and virtual reality.View more
School of Electrical and Electronics Engineering, Nanyang Technological University, Singapore
Jie Chen (S’13–M’16) received the B.S. and M.Eng. degrees from the School of Optical and Electronic Information, Huazhong University of Science and Technology, China, in 2008 and 2011, respectively, and the Ph.D. degree from the School of Electrical and Electronic Engineering, Nanyang Technological University (NTU), Singapore, in 2016. He is currently a Research Fellow with the ST Engineering-NTU Corporate Laboratory. His research interests are in computational photography (light field/hyperspectral signal compressed sensing and processing), computational and machine learning methods for image restoration (denoising, vision enhancement, and super resolution), and 3D content structure and motion analysis.
Jie Chen (S’13–M’16) received the B.S. and M.Eng. degrees from the School of Optical and Electronic Information, Huazhong University of Science and Technology, China, in 2008 and 2011, respectively, and the Ph.D. degree from the School of Electrical and Electronic Engineering, Nanyang Technological University (NTU), Singapore, in 2016. He is currently a Research Fellow with the ST Engineering-NTU Corporate Laboratory. His research interests are in computational photography (light field/hyperspectral signal compressed sensing and processing), computational and machine learning methods for image restoration (denoising, vision enhancement, and super resolution), and 3D content structure and motion analysis.View more
School of Information Science and Technology, University of Science and Technology of China, Hefei, China
Zhibo Chen (M’01–SM’11) received the B.Sc. and Ph.D. degrees from the Department of Electrical Engineering, Tsinghua University, in 1998 and 2003, respectively. He was with SONY and Thomson from 2003 to 2012. He was a Principal Scientist and a Research Manager at the Thomson Research & Innovation Department. He is currently a Professor with the University of Science and Technology of China. He has more than 50 granted and over 100 filed EU and U.S. patent applications and has more than 70 publications. His research interests include image and video compression, visual quality of experience assessment, immersive media computing, and intelligent media computing. He is a member of the IEEE Visual Signal Processing and Communications Committee and the IEEE Multimedia Communication Committee. He was an Organization Committee Member of ICIP 2017 and ICME 2013 and served as a TPC Member for the IEEE ISCAS and IEEE VCIP.
Zhibo Chen (M’01–SM’11) received the B.Sc. and Ph.D. degrees from the Department of Electrical Engineering, Tsinghua University, in 1998 and 2003, respectively. He was with SONY and Thomson from 2003 to 2012. He was a Principal Scientist and a Research Manager at the Thomson Research & Innovation Department. He is currently a Professor with the University of Science and Technology of China. He has more than 50 granted and over 100 filed EU and U.S. patent applications and has more than 70 publications. His research interests include image and video compression, visual quality of experience assessment, immersive media computing, and intelligent media computing. He is a member of the IEEE Visual Signal Processing and Communications Committee and the IEEE Multimedia Communication Committee. He was an Organization Committee Member of ICIP 2017 and ICME 2013 and served as a TPC Member for the IEEE ISCAS and IEEE VCIP.View more
School of Information Technologies, The University of Sydney, Sydney, NSW, Australia
Yuk Ying Chung received the B.Sc. degree in computing and information systems from the University of London in 1995 and the Ph.D. degree in computer engineering from the Queensland University of Technology, Australia, in 2000. From 1999 to 2001, she was a Lecturer at La Trobe University, Melbourne, Australia. Since 2001, she has been with the School of Information Technologies, The University of Sydney, Australia. Her research interests include image and video processing, virtual reality, deep neural network, AI, data mining, and big data analysis.
Yuk Ying Chung received the B.Sc. degree in computing and information systems from the University of London in 1995 and the Ph.D. degree in computer engineering from the Queensland University of Technology, Australia, in 2000. From 1999 to 2001, she was a Lecturer at La Trobe University, Melbourne, Australia. Since 2001, she has been with the School of Information Technologies, The University of Sydney, Australia. Her research interests include image and video processing, virtual reality, deep neural network, AI, data mining, and big data analysis.View more
