Abstract:
Recently, Transformer-based methods have shown impressive performance in single image super-resolution (SISR) tasks due to the ability of global feature extraction. Howev...Show MoreMetadata
Abstract:
Recently, Transformer-based methods have shown impressive performance in single image super-resolution (SISR) tasks due to the ability of global feature extraction. However, the capabilities of Transformers that need to incorporate contextual information to extract features dynamically are neglected. To address this issue, we propose a lightweight Cross-receptive Focused Inference Network (CFIN) that consists of a cascade of CT Blocks mixed with CNN and Transformer. Specifically, in the CT block, we first propose a CNN-based Cross-Scale Information Aggregation Module (CIAM) to enable the model to better focus on potentially helpful information to improve the efficiency of the Transformer phase. Then, we design a novel Cross-receptive Field Guided Transformer (CFGT) to enable the selection of contextual information required for reconstruction by using a modulated convolutional kernel that understands the current semantic information and exploits the information interaction within different self-attention. Extensive experiments have shown that our proposed CFIN can effectively reconstruct images using contextual information, and it can strike a good balance between computational cost and model performance as an efficient model.
Published in: IEEE Transactions on Multimedia ( Volume: 26)
Funding Agency:
Intelligent Visual Information Perception Laboratory, Institute of Advanced Technology, Nanjing University of Posts and Telecommunications, Nanjing, China
Provincial Key Laboratory for Computer Information Processing Technology, Soochow University, Suzhou, China
Wenjie Li received the B.S. degree in automation from Anhui Polytechnic University, Wuhu, China, in 2020. He is currently working toward the M.S. degree with the College of Automation and College of Artificial Intelligence, Nanjing University of Posts and Telecommunications, Nanjing, China. His research focuses on image restoration.
Wenjie Li received the B.S. degree in automation from Anhui Polytechnic University, Wuhu, China, in 2020. He is currently working toward the M.S. degree with the College of Automation and College of Artificial Intelligence, Nanjing University of Posts and Telecommunications, Nanjing, China. His research focuses on image restoration.View more
School of Communication and Information Engineering, Shanghai University, Shanghai, China
Jiangsu Key Laboratory of Image and Video Understanding for Social Safety, Nanjing University of Science and Technology, Nanjing, China
Juncheng Li received the Ph.D. degree in computer science and technology from East China Normal University, Shanghai, China, in 2021. He was a Postdoctoral Fellow with the Center for Mathematical Artificial Intelligence, The Chinese University of Hong Kong, Hong Kong. He is currently an Assistant Professor with the School of Communication and Information Engineering, Shanghai University, Shanghai. He has authored or coaut...Show More
Juncheng Li received the Ph.D. degree in computer science and technology from East China Normal University, Shanghai, China, in 2021. He was a Postdoctoral Fellow with the Center for Mathematical Artificial Intelligence, The Chinese University of Hong Kong, Hong Kong. He is currently an Assistant Professor with the School of Communication and Information Engineering, Shanghai University, Shanghai. He has authored or coaut...View more
Intelligent Visual Information Perception Laboratory, Institute of Advanced Technology, Nanjing University of Posts and Telecommunications, Nanjing, China
Provincial Key Laboratory for Computer Information Processing Technology, Soochow University, Suzhou, China
Guangwei Gao (Senior Member, IEEE) received the Ph.D. degree in pattern recognition and intelligence systems from the Nanjing University of Science and Technology, Nanjing, China, in 2014. He was a Visiting Student with the Department of Computing, The Hong Kong Polytechnic University, Hong Kong, in 2011 and 2013, respectively. From 2019 to 2021, he was a Project Researcher with the National Institute of Informatics, Toky...Show More
Guangwei Gao (Senior Member, IEEE) received the Ph.D. degree in pattern recognition and intelligence systems from the Nanjing University of Science and Technology, Nanjing, China, in 2014. He was a Visiting Student with the Department of Computing, The Hong Kong Polytechnic University, Hong Kong, in 2011 and 2013, respectively. From 2019 to 2021, he was a Project Researcher with the National Institute of Informatics, Toky...View more
Pattern Recognition and Intelligent System Laboratory, School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China
Weihong Deng (Member, IEEE) received the Ph.D. degree in signal and information processing from the Beijing University of Posts and Telecommunications (BUPT), Beijing, China, in 2009. He is currently a Professor with the School of Artificial Intelligence, BUPT. He has authored or coauthored more than 100 papers in international journals and conferences, such as IEEE Transactions on Pattern Analysis and Machine Intelligenc...Show More
Weihong Deng (Member, IEEE) received the Ph.D. degree in signal and information processing from the Beijing University of Posts and Telecommunications (BUPT), Beijing, China, in 2009. He is currently a Professor with the School of Artificial Intelligence, BUPT. He has authored or coauthored more than 100 papers in international journals and conferences, such as IEEE Transactions on Pattern Analysis and Machine Intelligenc...View more
State Key Laboratory of Internet of Things for Smart City, Faculty of Science and Technology, Department of Computer and Information Science, University of Macau, Macau, China
Jiantao Zhou (Senior Member, IEEE) received the Ph.D. degree from the Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology, Hong Kong, in 2009. He is currently an Associate Professor with the Faculty of Science and Technology, Department of Computer and Information Science, University of Macau, Macau, China. His research interests include multimedia security and forensics, and ...Show More
Jiantao Zhou (Senior Member, IEEE) received the Ph.D. degree from the Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology, Hong Kong, in 2009. He is currently an Associate Professor with the Faculty of Science and Technology, Department of Computer and Information Science, University of Macau, Macau, China. His research interests include multimedia security and forensics, and ...View more
School of Computer Science and Technology, Nanjing University of Science and Technology, Nanjing, China
Jian Yang (Member, IEEE) received the Ph.D. degree from the Nanjing University of Science and Technology (NJUST), Nanjing, China. From 2006 to 2007, he was a Postdoctoral Fellow with the Department of Computer Science, New Jersey Institute of Technology, Newark, NJ, USA. He is currently a Professor with the School of Computer Science and Technology, NJUST. He is the author of more than 400 scientific papers in pattern rec...Show More
Jian Yang (Member, IEEE) received the Ph.D. degree from the Nanjing University of Science and Technology (NJUST), Nanjing, China. From 2006 to 2007, he was a Postdoctoral Fellow with the Department of Computer Science, New Jersey Institute of Technology, Newark, NJ, USA. He is currently a Professor with the School of Computer Science and Technology, NJUST. He is the author of more than 400 scientific papers in pattern rec...View more
Research Center for Industries of the Future and the School of Engineering, Westlake University, Hangzhou, China
OPPO Research, Seattle, WA, USA
Guo-Jun Qi (Fellow, IEEE) is currently the Chief Scientist and the Director of the Seattle Research Center, OPPO Research USA, since 2021. Before that, he has been the Chief Scientist who led and oversaw an international R&D Team in the domain of multiple intelligent cloud services, including smart cities, visual computing services, medical intelligent services, and connected vehicle services with Futurewei since 2018. Si...Show More
Guo-Jun Qi (Fellow, IEEE) is currently the Chief Scientist and the Director of the Seattle Research Center, OPPO Research USA, since 2021. Before that, he has been the Chief Scientist who led and oversaw an international R&D Team in the domain of multiple intelligent cloud services, including smart cities, visual computing services, medical intelligent services, and connected vehicle services with Futurewei since 2018. Si...View more
Intelligent Visual Information Perception Laboratory, Institute of Advanced Technology, Nanjing University of Posts and Telecommunications, Nanjing, China
Provincial Key Laboratory for Computer Information Processing Technology, Soochow University, Suzhou, China
Wenjie Li received the B.S. degree in automation from Anhui Polytechnic University, Wuhu, China, in 2020. He is currently working toward the M.S. degree with the College of Automation and College of Artificial Intelligence, Nanjing University of Posts and Telecommunications, Nanjing, China. His research focuses on image restoration.
Wenjie Li received the B.S. degree in automation from Anhui Polytechnic University, Wuhu, China, in 2020. He is currently working toward the M.S. degree with the College of Automation and College of Artificial Intelligence, Nanjing University of Posts and Telecommunications, Nanjing, China. His research focuses on image restoration.View more
School of Communication and Information Engineering, Shanghai University, Shanghai, China
Jiangsu Key Laboratory of Image and Video Understanding for Social Safety, Nanjing University of Science and Technology, Nanjing, China
Juncheng Li received the Ph.D. degree in computer science and technology from East China Normal University, Shanghai, China, in 2021. He was a Postdoctoral Fellow with the Center for Mathematical Artificial Intelligence, The Chinese University of Hong Kong, Hong Kong. He is currently an Assistant Professor with the School of Communication and Information Engineering, Shanghai University, Shanghai. He has authored or coauthored more than 27 scientific papers in IEEE Transactions on Image Processing, IEEE Transactions on Neural Networks and Learning Systems, IEEE Transactions on Multimedia, International Conference on Computer Vision, European Conference on Computer Vision, Association for the Advancement of Artificial Intelligence, and International Joint Conference on Artificial Intelligence. His main research interests include image restoration, computer vision, and medical image processing.
Juncheng Li received the Ph.D. degree in computer science and technology from East China Normal University, Shanghai, China, in 2021. He was a Postdoctoral Fellow with the Center for Mathematical Artificial Intelligence, The Chinese University of Hong Kong, Hong Kong. He is currently an Assistant Professor with the School of Communication and Information Engineering, Shanghai University, Shanghai. He has authored or coauthored more than 27 scientific papers in IEEE Transactions on Image Processing, IEEE Transactions on Neural Networks and Learning Systems, IEEE Transactions on Multimedia, International Conference on Computer Vision, European Conference on Computer Vision, Association for the Advancement of Artificial Intelligence, and International Joint Conference on Artificial Intelligence. His main research interests include image restoration, computer vision, and medical image processing.View more
Intelligent Visual Information Perception Laboratory, Institute of Advanced Technology, Nanjing University of Posts and Telecommunications, Nanjing, China
Provincial Key Laboratory for Computer Information Processing Technology, Soochow University, Suzhou, China
Guangwei Gao (Senior Member, IEEE) received the Ph.D. degree in pattern recognition and intelligence systems from the Nanjing University of Science and Technology, Nanjing, China, in 2014. He was a Visiting Student with the Department of Computing, The Hong Kong Polytechnic University, Hong Kong, in 2011 and 2013, respectively. From 2019 to 2021, he was a Project Researcher with the National Institute of Informatics, Tokyo, Japan. He is currently an Associate Professor with the Nanjing University of Posts and Telecommunications, Nanjing. He has authored or coauthored more than 60 scientific papers in IEEE TIP/TCSVT/TITS/TMM/TIFS, ACM TOIT/TOMM, PR, AAAI, and IJCAI. His research interests include pattern recognition and computer vision.
Guangwei Gao (Senior Member, IEEE) received the Ph.D. degree in pattern recognition and intelligence systems from the Nanjing University of Science and Technology, Nanjing, China, in 2014. He was a Visiting Student with the Department of Computing, The Hong Kong Polytechnic University, Hong Kong, in 2011 and 2013, respectively. From 2019 to 2021, he was a Project Researcher with the National Institute of Informatics, Tokyo, Japan. He is currently an Associate Professor with the Nanjing University of Posts and Telecommunications, Nanjing. He has authored or coauthored more than 60 scientific papers in IEEE TIP/TCSVT/TITS/TMM/TIFS, ACM TOIT/TOMM, PR, AAAI, and IJCAI. His research interests include pattern recognition and computer vision.View more
Pattern Recognition and Intelligent System Laboratory, School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China
Weihong Deng (Member, IEEE) received the Ph.D. degree in signal and information processing from the Beijing University of Posts and Telecommunications (BUPT), Beijing, China, in 2009. He is currently a Professor with the School of Artificial Intelligence, BUPT. He has authored or coauthored more than 100 papers in international journals and conferences, such as IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Transactions on Image Processing, International Journal of Computer Vision, Conference on Computer Vision and Pattern Recognition, and International Conference on Computer Vision. His research interests include trustworthy biometrics and affective computing, with a particular emphasis on face recognition and expression analysis. He has been supported by the programs for New Century Excellent Talents and Young Changjiang Scholar by the Ministry of Education. His Dissertation was awarded the Outstanding Doctoral Dissertation Award by the Beijing Municipal Commission of Education.
Weihong Deng (Member, IEEE) received the Ph.D. degree in signal and information processing from the Beijing University of Posts and Telecommunications (BUPT), Beijing, China, in 2009. He is currently a Professor with the School of Artificial Intelligence, BUPT. He has authored or coauthored more than 100 papers in international journals and conferences, such as IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Transactions on Image Processing, International Journal of Computer Vision, Conference on Computer Vision and Pattern Recognition, and International Conference on Computer Vision. His research interests include trustworthy biometrics and affective computing, with a particular emphasis on face recognition and expression analysis. He has been supported by the programs for New Century Excellent Talents and Young Changjiang Scholar by the Ministry of Education. His Dissertation was awarded the Outstanding Doctoral Dissertation Award by the Beijing Municipal Commission of Education.View more
State Key Laboratory of Internet of Things for Smart City, Faculty of Science and Technology, Department of Computer and Information Science, University of Macau, Macau, China
Jiantao Zhou (Senior Member, IEEE) received the Ph.D. degree from the Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology, Hong Kong, in 2009. He is currently an Associate Professor with the Faculty of Science and Technology, Department of Computer and Information Science, University of Macau, Macau, China. His research interests include multimedia security and forensics, and multimedia signal processing. He has coauthored three articles that was the recipient of the Best Paper Award from the IEEE Pacific-Rim Conference on Multimedia in 2007, Best Student Paper Award, and Best Paper Runner-Up Award from the IEEE International Conference on Multimedia and Expo in 2016 and 2020. Since 2019, he has been an Associate Editor for the IEEE Transactions on Image Processing.
Jiantao Zhou (Senior Member, IEEE) received the Ph.D. degree from the Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology, Hong Kong, in 2009. He is currently an Associate Professor with the Faculty of Science and Technology, Department of Computer and Information Science, University of Macau, Macau, China. His research interests include multimedia security and forensics, and multimedia signal processing. He has coauthored three articles that was the recipient of the Best Paper Award from the IEEE Pacific-Rim Conference on Multimedia in 2007, Best Student Paper Award, and Best Paper Runner-Up Award from the IEEE International Conference on Multimedia and Expo in 2016 and 2020. Since 2019, he has been an Associate Editor for the IEEE Transactions on Image Processing.View more
School of Computer Science and Technology, Nanjing University of Science and Technology, Nanjing, China
Jian Yang (Member, IEEE) received the Ph.D. degree from the Nanjing University of Science and Technology (NJUST), Nanjing, China. From 2006 to 2007, he was a Postdoctoral Fellow with the Department of Computer Science, New Jersey Institute of Technology, Newark, NJ, USA. He is currently a Professor with the School of Computer Science and Technology, NJUST. He is the author of more than 400 scientific papers in pattern recognition and computer vision. His papers have been cited more than 28000 times in the Scholar Google. His research interests include pattern recognition, computer vision, and machine learning. He is/was an Associate Editor for Pattern Recognition and IEEE Transactions Neural Networks and Learning Systems. He is a Fellow of IAPR.
Jian Yang (Member, IEEE) received the Ph.D. degree from the Nanjing University of Science and Technology (NJUST), Nanjing, China. From 2006 to 2007, he was a Postdoctoral Fellow with the Department of Computer Science, New Jersey Institute of Technology, Newark, NJ, USA. He is currently a Professor with the School of Computer Science and Technology, NJUST. He is the author of more than 400 scientific papers in pattern recognition and computer vision. His papers have been cited more than 28000 times in the Scholar Google. His research interests include pattern recognition, computer vision, and machine learning. He is/was an Associate Editor for Pattern Recognition and IEEE Transactions Neural Networks and Learning Systems. He is a Fellow of IAPR.View more
Research Center for Industries of the Future and the School of Engineering, Westlake University, Hangzhou, China
OPPO Research, Seattle, WA, USA
Guo-Jun Qi (Fellow, IEEE) is currently the Chief Scientist and the Director of the Seattle Research Center, OPPO Research USA, since 2021. Before that, he has been the Chief Scientist who led and oversaw an international R&D Team in the domain of multiple intelligent cloud services, including smart cities, visual computing services, medical intelligent services, and connected vehicle services with Futurewei since 2018. Since 2014, he was a Faculty Member of the Department of Computer Science and the Director of the MAchine Perception and LEarning (MAPLE) Lab, University of Central Florida, Orlando, FL, USA. Prior to that, he was also a Research Staff Member with IBM T. J. Watson Research Center, Yorktown Heights, NY, USA.
Guo-Jun Qi (Fellow, IEEE) is currently the Chief Scientist and the Director of the Seattle Research Center, OPPO Research USA, since 2021. Before that, he has been the Chief Scientist who led and oversaw an international R&D Team in the domain of multiple intelligent cloud services, including smart cities, visual computing services, medical intelligent services, and connected vehicle services with Futurewei since 2018. Since 2014, he was a Faculty Member of the Department of Computer Science and the Director of the MAchine Perception and LEarning (MAPLE) Lab, University of Central Florida, Orlando, FL, USA. Prior to that, he was also a Research Staff Member with IBM T. J. Watson Research Center, Yorktown Heights, NY, USA.View more