Going the Extra Mile in Face Image Quality Assessment: A Novel Database and Model | IEEE Journals & Magazine | IEEE Xplore

Going the Extra Mile in Face Image Quality Assessment: A Novel Database and Model


Abstract:

An accurate computational model for image quality assessment (IQA) benefits many vision applications, such as image filtering, image processing, and image generation. Alt...Show More

Abstract:

An accurate computational model for image quality assessment (IQA) benefits many vision applications, such as image filtering, image processing, and image generation. Although the study of face images is an important subfield in computer vision research, the lack of face IQA data and models limits the precision of current IQA metrics on face image processing tasks such as face superresolution, face enhancement, and face editing. To narrow this gap, in this article, we first introduce the largest annotated IQA database developed to date, which contains 20,000 human faces – an order of magnitude larger than all existing rated datasets of faces – of diverse individuals in highly varied circumstances. Based on the database, we further propose a novel deep learning model to accurately predict face image quality, which, for the first time, explores the use of generative priors for IQA. By taking advantage of rich statistics encoded in well pretrained off-the-shelf generative models, we obtain generative prior information and use it as latent references to facilitate blind IQA. The experimental results demonstrate both the value of the proposed dataset for face IQA and the superior performance of the proposed model.
Published in: IEEE Transactions on Multimedia ( Volume: 26)
Page(s): 2671 - 2685
Date of Publication: 04 August 2023

ISSN Information:

Funding Agency:

Author image of Shaolin Su
School of Computer Science and Engineering, Northwestern Polytechnical University, Xi An Shi, China
Department of Computer and Information Science, University of Konstanz, Konstanz, Germany
Shaolin Su received the M.S. degree from the Harbin Institute of Technology, Harbin, China in 2014, and the Ph.D. degree from Northwestern Polytechnical University, Xi'an, China in 2023. He is currently a Visiting Researcher with the University of Konstanz, Germany, Konstanz, Germany. His research interests include image quality assessment, image restoration, and image synthesis.
Shaolin Su received the M.S. degree from the Harbin Institute of Technology, Harbin, China in 2014, and the Ph.D. degree from Northwestern Polytechnical University, Xi'an, China in 2023. He is currently a Visiting Researcher with the University of Konstanz, Germany, Konstanz, Germany. His research interests include image quality assessment, image restoration, and image synthesis.View more
Author image of Hanhe Lin
School of Science and Engineering, University of Dundee, Dundee, U.K.
Department of Computer and Information Science, University of Konstanz, Konstanz, Germany
Hanhe Lin received the Ph.D. degree from the Department of Information Science, University of Otago, Dunedin, New Zealand, in 2016. From 2016 to 2021, he was a Postdoc at the Department of Computer and Information Science, University of Konstanz, Konstanz, Germany, where he was working on project A05 (visual quality assessment) of SFB-TRR 161, funded by the German Research Foundation (DFG). He is currently a Lecturer in c...Show More
Hanhe Lin received the Ph.D. degree from the Department of Information Science, University of Otago, Dunedin, New Zealand, in 2016. From 2016 to 2021, he was a Postdoc at the Department of Computer and Information Science, University of Konstanz, Konstanz, Germany, where he was working on project A05 (visual quality assessment) of SFB-TRR 161, funded by the German Research Foundation (DFG). He is currently a Lecturer in c...View more
Author image of Vlad Hosu
Department of Computer and Information Science, University of Konstanz, Konstanz, Germany
Vlad Hosu received the Ph.D. degree in computer vision from the National University of Singapore, Singapore, with the dissertation on computational aesthetics in photography. He explores the intersection of technical and aesthetic quality assessment by studying human and machine visual perception. He is developing reliable visual quality models using deep learning and crowdsourcing. His contributions include co-authoring ...Show More
Vlad Hosu received the Ph.D. degree in computer vision from the National University of Singapore, Singapore, with the dissertation on computational aesthetics in photography. He explores the intersection of technical and aesthetic quality assessment by studying human and machine visual perception. He is developing reliable visual quality models using deep learning and crowdsourcing. His contributions include co-authoring ...View more
Author image of Oliver Wiedemann
Department of Computer and Information Science, University of Konstanz, Konstanz, Germany
Oliver Wiedemann received the B.Sc. and M.Sc. degrees in computer and information science from the University of Konstanz, Konstanz, Germany, in 2018 and 2020. He is currently working toward the Ph.D. degree working in the field of image quality assessment. His research interests include no-reference metrics in the context of scale-invariance, processing of high-resolution images and unsupervised representation learning.
Oliver Wiedemann received the B.Sc. and M.Sc. degrees in computer and information science from the University of Konstanz, Konstanz, Germany, in 2018 and 2020. He is currently working toward the Ph.D. degree working in the field of image quality assessment. His research interests include no-reference metrics in the context of scale-invariance, processing of high-resolution images and unsupervised representation learning.View more
Author image of Jinqiu Sun
School of Computer Science and Engineering, Northwestern Polytechnical University, Xi An Shi, China
Jinqiu Sun received the B.S. and the M.S. and Ph.D. degrees from Northwestern Polytechnical University, Xi'an, China, in 1999, from Northwestern Polytechnical University in 2004 and 2005 respectively. She is currently a Professor of the School of Astronomy, Northwestern Polytechnical University. Her research interests include signal and image processing, computer vision, and pattern recognition.
Jinqiu Sun received the B.S. and the M.S. and Ph.D. degrees from Northwestern Polytechnical University, Xi'an, China, in 1999, from Northwestern Polytechnical University in 2004 and 2005 respectively. She is currently a Professor of the School of Astronomy, Northwestern Polytechnical University. Her research interests include signal and image processing, computer vision, and pattern recognition.View more
Author image of Yu Zhu
School of Computer Science and Engineering, Northwestern Polytechnical University, Xi An Shi, China
Yu Zhu received the Ph.D. degrees from Northwestern Polytechnical University (NPU), Xi'an, China, in 2017. He is currently an Assistant Researcher with the School of Computer Science, NPU. His research interests include image debluring, Image enhancement, and image super-resolution.
Yu Zhu received the Ph.D. degrees from Northwestern Polytechnical University (NPU), Xi'an, China, in 2017. He is currently an Assistant Researcher with the School of Computer Science, NPU. His research interests include image debluring, Image enhancement, and image super-resolution.View more
Author image of Hantao Liu
School of Computer Science and Informatics, Cardiff University, Cardiff, U.K.
Hantao Liu received the Ph.D. degree from the Delft University of Technology, Delft, The Netherlands, in 2011. He is currently an Associate Professor with the School of Computer Science and Informatics, Cardiff University, Cardiff, U.K. He is currently an Associate Editor for IEEE Transactions on Circuits and Systems for Video Technology and IEEE Signal Processing Letters.
Hantao Liu received the Ph.D. degree from the Delft University of Technology, Delft, The Netherlands, in 2011. He is currently an Associate Professor with the School of Computer Science and Informatics, Cardiff University, Cardiff, U.K. He is currently an Associate Editor for IEEE Transactions on Circuits and Systems for Video Technology and IEEE Signal Processing Letters.View more
Author image of Yanning Zhang
School of Computer Science and Engineering, Northwestern Polytechnical University, Xi An Shi, China
Yanning Zhang received the B.S. degree from the Dalian University of Technology, Dalian, China, in 1988, the M.S. degree from the School of Electronic Engineering, Northwestern Polytechnical University, Xi'an, China, in 1993, and the Ph.D. degree from the School of Marine Engineering, Northwestern Polytechnical University, in 1996. She is currently a Professor with the School of Computer Science, Northwestern Polytechnica...Show More
Yanning Zhang received the B.S. degree from the Dalian University of Technology, Dalian, China, in 1988, the M.S. degree from the School of Electronic Engineering, Northwestern Polytechnical University, Xi'an, China, in 1993, and the Ph.D. degree from the School of Marine Engineering, Northwestern Polytechnical University, in 1996. She is currently a Professor with the School of Computer Science, Northwestern Polytechnica...View more
Author image of Dietmar Saupe
Department of Computer and Information Science, University of Konstanz, Konstanz, Germany
Dietmar Saupe was born in Bremen, Germany, in 1954. He received the Dr.rer.nat. degree in mathematics from the University of Bremen, Bremen, Germany, in 1982. From 1985 to 1993, he was an Assistant Professor with the Departments of Mathematics, first with the University of California, Santa Cruz, CA, USA, and then with the University of Bremen, Bremen, Germany, resulting in his habilitation. From 1993 to 1998, he was a Pr...Show More
Dietmar Saupe was born in Bremen, Germany, in 1954. He received the Dr.rer.nat. degree in mathematics from the University of Bremen, Bremen, Germany, in 1982. From 1985 to 1993, he was an Assistant Professor with the Departments of Mathematics, first with the University of California, Santa Cruz, CA, USA, and then with the University of Bremen, Bremen, Germany, resulting in his habilitation. From 1993 to 1998, he was a Pr...View more

Author image of Shaolin Su
School of Computer Science and Engineering, Northwestern Polytechnical University, Xi An Shi, China
Department of Computer and Information Science, University of Konstanz, Konstanz, Germany
Shaolin Su received the M.S. degree from the Harbin Institute of Technology, Harbin, China in 2014, and the Ph.D. degree from Northwestern Polytechnical University, Xi'an, China in 2023. He is currently a Visiting Researcher with the University of Konstanz, Germany, Konstanz, Germany. His research interests include image quality assessment, image restoration, and image synthesis.
Shaolin Su received the M.S. degree from the Harbin Institute of Technology, Harbin, China in 2014, and the Ph.D. degree from Northwestern Polytechnical University, Xi'an, China in 2023. He is currently a Visiting Researcher with the University of Konstanz, Germany, Konstanz, Germany. His research interests include image quality assessment, image restoration, and image synthesis.View more
Author image of Hanhe Lin
School of Science and Engineering, University of Dundee, Dundee, U.K.
Department of Computer and Information Science, University of Konstanz, Konstanz, Germany
Hanhe Lin received the Ph.D. degree from the Department of Information Science, University of Otago, Dunedin, New Zealand, in 2016. From 2016 to 2021, he was a Postdoc at the Department of Computer and Information Science, University of Konstanz, Konstanz, Germany, where he was working on project A05 (visual quality assessment) of SFB-TRR 161, funded by the German Research Foundation (DFG). He is currently a Lecturer in computing with the University of Dundee, Dundee, U.K. His research interests include image processing, computer vision, machine learning, deep learning, and visual quality assessment. He is a member of technical program committee or a reviewer in a number of conferences/journals, e.g. QoMEX, IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Transactions on Image Processing.
Hanhe Lin received the Ph.D. degree from the Department of Information Science, University of Otago, Dunedin, New Zealand, in 2016. From 2016 to 2021, he was a Postdoc at the Department of Computer and Information Science, University of Konstanz, Konstanz, Germany, where he was working on project A05 (visual quality assessment) of SFB-TRR 161, funded by the German Research Foundation (DFG). He is currently a Lecturer in computing with the University of Dundee, Dundee, U.K. His research interests include image processing, computer vision, machine learning, deep learning, and visual quality assessment. He is a member of technical program committee or a reviewer in a number of conferences/journals, e.g. QoMEX, IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Transactions on Image Processing.View more
Author image of Vlad Hosu
Department of Computer and Information Science, University of Konstanz, Konstanz, Germany
Vlad Hosu received the Ph.D. degree in computer vision from the National University of Singapore, Singapore, with the dissertation on computational aesthetics in photography. He explores the intersection of technical and aesthetic quality assessment by studying human and machine visual perception. He is developing reliable visual quality models using deep learning and crowdsourcing. His contributions include co-authoring several databases central to the field and significantly improving state-of-the-art predictive models, leading to many applications.
Vlad Hosu received the Ph.D. degree in computer vision from the National University of Singapore, Singapore, with the dissertation on computational aesthetics in photography. He explores the intersection of technical and aesthetic quality assessment by studying human and machine visual perception. He is developing reliable visual quality models using deep learning and crowdsourcing. His contributions include co-authoring several databases central to the field and significantly improving state-of-the-art predictive models, leading to many applications.View more
Author image of Oliver Wiedemann
Department of Computer and Information Science, University of Konstanz, Konstanz, Germany
Oliver Wiedemann received the B.Sc. and M.Sc. degrees in computer and information science from the University of Konstanz, Konstanz, Germany, in 2018 and 2020. He is currently working toward the Ph.D. degree working in the field of image quality assessment. His research interests include no-reference metrics in the context of scale-invariance, processing of high-resolution images and unsupervised representation learning.
Oliver Wiedemann received the B.Sc. and M.Sc. degrees in computer and information science from the University of Konstanz, Konstanz, Germany, in 2018 and 2020. He is currently working toward the Ph.D. degree working in the field of image quality assessment. His research interests include no-reference metrics in the context of scale-invariance, processing of high-resolution images and unsupervised representation learning.View more
Author image of Jinqiu Sun
School of Computer Science and Engineering, Northwestern Polytechnical University, Xi An Shi, China
Jinqiu Sun received the B.S. and the M.S. and Ph.D. degrees from Northwestern Polytechnical University, Xi'an, China, in 1999, from Northwestern Polytechnical University in 2004 and 2005 respectively. She is currently a Professor of the School of Astronomy, Northwestern Polytechnical University. Her research interests include signal and image processing, computer vision, and pattern recognition.
Jinqiu Sun received the B.S. and the M.S. and Ph.D. degrees from Northwestern Polytechnical University, Xi'an, China, in 1999, from Northwestern Polytechnical University in 2004 and 2005 respectively. She is currently a Professor of the School of Astronomy, Northwestern Polytechnical University. Her research interests include signal and image processing, computer vision, and pattern recognition.View more
Author image of Yu Zhu
School of Computer Science and Engineering, Northwestern Polytechnical University, Xi An Shi, China
Yu Zhu received the Ph.D. degrees from Northwestern Polytechnical University (NPU), Xi'an, China, in 2017. He is currently an Assistant Researcher with the School of Computer Science, NPU. His research interests include image debluring, Image enhancement, and image super-resolution.
Yu Zhu received the Ph.D. degrees from Northwestern Polytechnical University (NPU), Xi'an, China, in 2017. He is currently an Assistant Researcher with the School of Computer Science, NPU. His research interests include image debluring, Image enhancement, and image super-resolution.View more
Author image of Hantao Liu
School of Computer Science and Informatics, Cardiff University, Cardiff, U.K.
Hantao Liu received the Ph.D. degree from the Delft University of Technology, Delft, The Netherlands, in 2011. He is currently an Associate Professor with the School of Computer Science and Informatics, Cardiff University, Cardiff, U.K. He is currently an Associate Editor for IEEE Transactions on Circuits and Systems for Video Technology and IEEE Signal Processing Letters.
Hantao Liu received the Ph.D. degree from the Delft University of Technology, Delft, The Netherlands, in 2011. He is currently an Associate Professor with the School of Computer Science and Informatics, Cardiff University, Cardiff, U.K. He is currently an Associate Editor for IEEE Transactions on Circuits and Systems for Video Technology and IEEE Signal Processing Letters.View more
Author image of Yanning Zhang
School of Computer Science and Engineering, Northwestern Polytechnical University, Xi An Shi, China
Yanning Zhang received the B.S. degree from the Dalian University of Technology, Dalian, China, in 1988, the M.S. degree from the School of Electronic Engineering, Northwestern Polytechnical University, Xi'an, China, in 1993, and the Ph.D. degree from the School of Marine Engineering, Northwestern Polytechnical University, in 1996. She is currently a Professor with the School of Computer Science, Northwestern Polytechnical University. She is also a Cheung Kong Professor of Ministry of Education, China. Her current research interests include remote sensing image analysis, computer vision, and pattern recognition.
Yanning Zhang received the B.S. degree from the Dalian University of Technology, Dalian, China, in 1988, the M.S. degree from the School of Electronic Engineering, Northwestern Polytechnical University, Xi'an, China, in 1993, and the Ph.D. degree from the School of Marine Engineering, Northwestern Polytechnical University, in 1996. She is currently a Professor with the School of Computer Science, Northwestern Polytechnical University. She is also a Cheung Kong Professor of Ministry of Education, China. Her current research interests include remote sensing image analysis, computer vision, and pattern recognition.View more
Author image of Dietmar Saupe
Department of Computer and Information Science, University of Konstanz, Konstanz, Germany
Dietmar Saupe was born in Bremen, Germany, in 1954. He received the Dr.rer.nat. degree in mathematics from the University of Bremen, Bremen, Germany, in 1982. From 1985 to 1993, he was an Assistant Professor with the Departments of Mathematics, first with the University of California, Santa Cruz, CA, USA, and then with the University of Bremen, Bremen, Germany, resulting in his habilitation. From 1993 to 1998, he was a Professor of computer science with the University of Freiburg, Breisgau, Germany, the University of Leipzig, Leipzig, Germany, till 2002, and since then, the University of Konstanz, Konstanz, Germany. He is the coauthor of the book Chaos and Fractals (Springer-Verlag, 1992). His research interests include image and video processing, computer graphics, scientific visualisation, dynamical systems, and sport informatics. His coauthor book Chaos and Fractals (Springer-Verlag, 1992) was the recipient of the Association of American Publishers Award for Best Mathematics Book of the Year, the book The Science of Fractal Images (Springer-Verlag, 1988), and well over 100 research articles.
Dietmar Saupe was born in Bremen, Germany, in 1954. He received the Dr.rer.nat. degree in mathematics from the University of Bremen, Bremen, Germany, in 1982. From 1985 to 1993, he was an Assistant Professor with the Departments of Mathematics, first with the University of California, Santa Cruz, CA, USA, and then with the University of Bremen, Bremen, Germany, resulting in his habilitation. From 1993 to 1998, he was a Professor of computer science with the University of Freiburg, Breisgau, Germany, the University of Leipzig, Leipzig, Germany, till 2002, and since then, the University of Konstanz, Konstanz, Germany. He is the coauthor of the book Chaos and Fractals (Springer-Verlag, 1992). His research interests include image and video processing, computer graphics, scientific visualisation, dynamical systems, and sport informatics. His coauthor book Chaos and Fractals (Springer-Verlag, 1992) was the recipient of the Association of American Publishers Award for Best Mathematics Book of the Year, the book The Science of Fractal Images (Springer-Verlag, 1988), and well over 100 research articles.View more

References

References is not available for this document.