Abstract:
Multi-human parsing is an image segmentation task necessitating both instance-level and fine-grained category-level information. However, prior research has typically pro...Show MoreMetadata
Abstract:
Multi-human parsing is an image segmentation task necessitating both instance-level and fine-grained category-level information. However, prior research has typically processed these two types of information through distinct branch types and output formats, leading to inefficient and redundant frameworks. This paper introduces UniParser, which integrates instance-level and category-level representations in three key aspects: 1) we propose a unified correlation representation learning approach, allowing our network to learn instance and category features within the cosine space; 2) we unify the form of outputs of each modules as pixel-level results while supervising instance and category features using a homogeneous label accompanied by an auxiliary loss; and 3) we design a joint optimization procedure to fuse instance and category representations. By unifying instance-level and category-level output, UniParser circumvents manually designed post-processing techniques and surpasses state-of-the-art methods, achieving 49.3% AP on MHPv2.0 and 60.4% AP on CIHP. We have released our source code, pretrained models, and demos to facilitate future studies on https://github.com/cjm-sfw/Uniparser.
Published in: IEEE Transactions on Image Processing ( Volume: 33)
Funding Agency:
School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing, China
Jiaming Chu is currently pursuing the Ph.D. degree in electronic science and technology with Beijing University of Posts and Telecommunications. His research interests include deep learning and computer vision, with in-depth research in sub-fields, such as human action recognition, instance segmentation, and human parsing.
Jiaming Chu is currently pursuing the Ph.D. degree in electronic science and technology with Beijing University of Posts and Telecommunications. His research interests include deep learning and computer vision, with in-depth research in sub-fields, such as human action recognition, instance segmentation, and human parsing.View more
School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing, China
Lei Jin received the degree from Beijing University of Posts and Telecommunications (BUPT), Beijing, China. He is currently an Associate Research Fellow with BUPT. His research interests include computer vision, data mining, and pattern recognition, with in-depth research in sub-fields, such as human pose estimation, human action recognition, and human parsing, with related research results published in high-level confere...Show More
Lei Jin received the degree from Beijing University of Posts and Telecommunications (BUPT), Beijing, China. He is currently an Associate Research Fellow with BUPT. His research interests include computer vision, data mining, and pattern recognition, with in-depth research in sub-fields, such as human pose estimation, human action recognition, and human parsing, with related research results published in high-level confere...View more
School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing, China
Yinglei Teng (Senior Member, IEEE) received the B.S. degree from Shandong University, China, in 2005, and the Ph.D. degree in electrical engineering from Beijing University of Posts and Telecommunications (BUPT), in 2011. She is currently a Professor with the School of Electronic Engineering, BUPT. Her current research interests include wireless communications, edge intelligence, and connected AI.
Yinglei Teng (Senior Member, IEEE) received the B.S. degree from Shandong University, China, in 2005, and the Ph.D. degree in electrical engineering from Beijing University of Posts and Telecommunications (BUPT), in 2011. She is currently a Professor with the School of Electronic Engineering, BUPT. Her current research interests include wireless communications, edge intelligence, and connected AI.View more
Ant Group, Beijing, China
Jianshu Li received the Ph.D. degree from the School of Computing, National University of Singapore, in 2019, advised by Prof. Terence Sim and Prof. Shuicheng Yan. He has been an Algorithm Expert with Ant Group since 2018, mainly working on face analysis algorithms, including face recognition, face liveness detection, face quality analysis, and Deepfake detection. He has published more than 20 papers in journals and confe...Show More
Jianshu Li received the Ph.D. degree from the School of Computing, National University of Singapore, in 2019, advised by Prof. Terence Sim and Prof. Shuicheng Yan. He has been an Algorithm Expert with Ant Group since 2018, mainly working on face analysis algorithms, including face recognition, face liveness detection, face quality analysis, and Deepfake detection. He has published more than 20 papers in journals and confe...View more
Beijing Jiaotong University, Beijing, China
Yunchao Wei is currently a Full Professor with Beijing Jiaotong University. He has published more than 100 papers in top-tier conferences/journals and more than Google citations 13000. He has broad research interests in computer vision and machine learning. His current research interests include visual recognition with imperfect data, image/video segmentation and object detection, and multi-modal perception. He was select...Show More
Yunchao Wei is currently a Full Professor with Beijing Jiaotong University. He has published more than 100 papers in top-tier conferences/journals and more than Google citations 13000. He has broad research interests in computer vision and machine learning. His current research interests include visual recognition with imperfect data, image/video segmentation and object detection, and multi-modal perception. He was select...View more
Wuhan University, Wuhan, China
Zheng Wang (Senior Member, IEEE) received the Ph.D. degree from the National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, in 2017. He was a JSPS Fellowship Researcher with the National Institute of Informatics, Japan, and a Project Assistant Professor with The University of Tokyo, Japan. He is currently a Professor with the National Engineering Research Center for Mult...Show More
Zheng Wang (Senior Member, IEEE) received the Ph.D. degree from the National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, in 2017. He was a JSPS Fellowship Researcher with the National Institute of Informatics, Japan, and a Project Assistant Professor with The University of Tokyo, Japan. He is currently a Professor with the National Engineering Research Center for Mult...View more
Tsinghua University, Beijing, China
Junliang Xing (Senior Member, IEEE) received the dual B.E. degree in computer science and applied mathematics from Xi’an Jiaotong University in 2007 and the Ph.D. degree in computer science and technology from Tsinghua University in 2012. He is currently a Professor with the Department of Computer Science and Technology, Tsinghua University. He has published over 120 peer-reviewed conference papers, such as IJCAI, AAAI, I...Show More
Junliang Xing (Senior Member, IEEE) received the dual B.E. degree in computer science and applied mathematics from Xi’an Jiaotong University in 2007 and the Ph.D. degree in computer science and technology from Tsinghua University in 2012. He is currently a Professor with the Department of Computer Science and Technology, Tsinghua University. He has published over 120 peer-reviewed conference papers, such as IJCAI, AAAI, I...View more
Skywork AI, 2 Science park drive, Singapore
Shuicheng Yan (Fellow, IEEE) is currently the Director of Skywork AI, Singapore. He has authored or co-authored more than 600 papers in top international journals and conferences, with Google Scholar Citation more than 40000 times, and H-index of 105. His research interests include computer vision, machine learning, and multimedia analysis. He is a fellow of the Academy of Engineering, Singapore, an ACM Fellow, and an IAP...Show More
Shuicheng Yan (Fellow, IEEE) is currently the Director of Skywork AI, Singapore. He has authored or co-authored more than 600 papers in top international journals and conferences, with Google Scholar Citation more than 40000 times, and H-index of 105. His research interests include computer vision, machine learning, and multimedia analysis. He is a fellow of the Academy of Engineering, Singapore, an ACM Fellow, and an IAP...View more
EVOL Lab, Institute of AI (TeleAI), China Telecom and the School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University (NWPU), Xi’an, China
Jian Zhao (Member, IEEE) received the Ph.D. degree from the National University of Singapore (NUS). He is currently the Leader of the EVOL Laboratory, a Principal Research Scientist with the Institute of AI (TeleAI), China Telecom, China, and a Researcher and the Ph.D. Supervisor of the School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University (NWPU), Xi’an, Shanxi, China. He...Show More
Jian Zhao (Member, IEEE) received the Ph.D. degree from the National University of Singapore (NUS). He is currently the Leader of the EVOL Laboratory, a Principal Research Scientist with the Institute of AI (TeleAI), China Telecom, China, and a Researcher and the Ph.D. Supervisor of the School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University (NWPU), Xi’an, Shanxi, China. He...View more
Contents School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing, China
Jiaming Chu is currently pursuing the Ph.D. degree in electronic science and technology with Beijing University of Posts and Telecommunications. His research interests include deep learning and computer vision, with in-depth research in sub-fields, such as human action recognition, instance segmentation, and human parsing.
Jiaming Chu is currently pursuing the Ph.D. degree in electronic science and technology with Beijing University of Posts and Telecommunications. His research interests include deep learning and computer vision, with in-depth research in sub-fields, such as human action recognition, instance segmentation, and human parsing.View more
School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing, China
Lei Jin received the degree from Beijing University of Posts and Telecommunications (BUPT), Beijing, China. He is currently an Associate Research Fellow with BUPT. His research interests include computer vision, data mining, and pattern recognition, with in-depth research in sub-fields, such as human pose estimation, human action recognition, and human parsing, with related research results published in high-level conferences and journals, such as CVPR, AAAI, NIPS, IJCAI, and ACMMM.
Lei Jin received the degree from Beijing University of Posts and Telecommunications (BUPT), Beijing, China. He is currently an Associate Research Fellow with BUPT. His research interests include computer vision, data mining, and pattern recognition, with in-depth research in sub-fields, such as human pose estimation, human action recognition, and human parsing, with related research results published in high-level conferences and journals, such as CVPR, AAAI, NIPS, IJCAI, and ACMMM.View more
School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing, China
Yinglei Teng (Senior Member, IEEE) received the B.S. degree from Shandong University, China, in 2005, and the Ph.D. degree in electrical engineering from Beijing University of Posts and Telecommunications (BUPT), in 2011. She is currently a Professor with the School of Electronic Engineering, BUPT. Her current research interests include wireless communications, edge intelligence, and connected AI.
Yinglei Teng (Senior Member, IEEE) received the B.S. degree from Shandong University, China, in 2005, and the Ph.D. degree in electrical engineering from Beijing University of Posts and Telecommunications (BUPT), in 2011. She is currently a Professor with the School of Electronic Engineering, BUPT. Her current research interests include wireless communications, edge intelligence, and connected AI.View more
Ant Group, Beijing, China
Jianshu Li received the Ph.D. degree from the School of Computing, National University of Singapore, in 2019, advised by Prof. Terence Sim and Prof. Shuicheng Yan. He has been an Algorithm Expert with Ant Group since 2018, mainly working on face analysis algorithms, including face recognition, face liveness detection, face quality analysis, and Deepfake detection. He has published more than 20 papers in journals and conferences. His research interests include computer vision and image understanding, particularly face and human analytics, semantic segmentation, and object detection. He is the winner of the Gold Award of PREMIA 2019 Singapore, the Best Student Paper Award of ACMMM 2018, winner prize of object localization ILSVRC 2017, and winner prize of emotion recognition challenge ICMI 2016. He served as an Invited Reviewer for CVPR, ECCV, NIPS, IJCAI, FG, ICMI, IEEE Transactions on Image Processing, IEEE Transactions on Circuits and Systems for Video Technology, and IEEE Transactions on Multimedia.
Jianshu Li received the Ph.D. degree from the School of Computing, National University of Singapore, in 2019, advised by Prof. Terence Sim and Prof. Shuicheng Yan. He has been an Algorithm Expert with Ant Group since 2018, mainly working on face analysis algorithms, including face recognition, face liveness detection, face quality analysis, and Deepfake detection. He has published more than 20 papers in journals and conferences. His research interests include computer vision and image understanding, particularly face and human analytics, semantic segmentation, and object detection. He is the winner of the Gold Award of PREMIA 2019 Singapore, the Best Student Paper Award of ACMMM 2018, winner prize of object localization ILSVRC 2017, and winner prize of emotion recognition challenge ICMI 2016. He served as an Invited Reviewer for CVPR, ECCV, NIPS, IJCAI, FG, ICMI, IEEE Transactions on Image Processing, IEEE Transactions on Circuits and Systems for Video Technology, and IEEE Transactions on Multimedia.View more
Beijing Jiaotong University, Beijing, China
Yunchao Wei is currently a Full Professor with Beijing Jiaotong University. He has published more than 100 papers in top-tier conferences/journals and more than Google citations 13000. He has broad research interests in computer vision and machine learning. His current research interests include visual recognition with imperfect data, image/video segmentation and object detection, and multi-modal perception. He was selected as a MIT TR35 China by MIT Technology Review in 2021 and was named as one of the five top early-career researchers in engineering and computer sciences in Australia by Australian in 2020. He received the Discovery Early Career Researcher Award by Australian Research Council in 2019 and the First Prize in Science and Technology awarded by China Society of Image and Graphics (CSIG) in 2019. He received many competition prizes from CVPR/ICCV/ECCV, such as the winner prizes of ILSVRC 2014, LIP 2018/2019, and Youtube VOS 2021; and runner-up prizes of ILSVRC 2017 and DAVIS 2020. He organized many workshops on top-tier conferences, including Learning from Imperfect Data Workshop series (CVPR 2019, 2020, and 2021) and Real-world Recognition from Low-Quality Inputs Workshop series (ICCV 2019 and ECCV 2020).
Yunchao Wei is currently a Full Professor with Beijing Jiaotong University. He has published more than 100 papers in top-tier conferences/journals and more than Google citations 13000. He has broad research interests in computer vision and machine learning. His current research interests include visual recognition with imperfect data, image/video segmentation and object detection, and multi-modal perception. He was selected as a MIT TR35 China by MIT Technology Review in 2021 and was named as one of the five top early-career researchers in engineering and computer sciences in Australia by Australian in 2020. He received the Discovery Early Career Researcher Award by Australian Research Council in 2019 and the First Prize in Science and Technology awarded by China Society of Image and Graphics (CSIG) in 2019. He received many competition prizes from CVPR/ICCV/ECCV, such as the winner prizes of ILSVRC 2014, LIP 2018/2019, and Youtube VOS 2021; and runner-up prizes of ILSVRC 2017 and DAVIS 2020. He organized many workshops on top-tier conferences, including Learning from Imperfect Data Workshop series (CVPR 2019, 2020, and 2021) and Real-world Recognition from Low-Quality Inputs Workshop series (ICCV 2019 and ECCV 2020).View more
Wuhan University, Wuhan, China
Zheng Wang (Senior Member, IEEE) received the Ph.D. degree from the National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, in 2017. He was a JSPS Fellowship Researcher with the National Institute of Informatics, Japan, and a Project Assistant Professor with The University of Tokyo, Japan. He is currently a Professor with the National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University. His research interests include image processing and video content analysis. He is also an Associate Editor of IEEE Transactions on Image Processing.
Zheng Wang (Senior Member, IEEE) received the Ph.D. degree from the National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, in 2017. He was a JSPS Fellowship Researcher with the National Institute of Informatics, Japan, and a Project Assistant Professor with The University of Tokyo, Japan. He is currently a Professor with the National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University. His research interests include image processing and video content analysis. He is also an Associate Editor of IEEE Transactions on Image Processing.View more
Tsinghua University, Beijing, China
Junliang Xing (Senior Member, IEEE) received the dual B.E. degree in computer science and applied mathematics from Xi’an Jiaotong University in 2007 and the Ph.D. degree in computer science and technology from Tsinghua University in 2012. He is currently a Professor with the Department of Computer Science and Technology, Tsinghua University. He has published over 120 peer-reviewed conference papers, such as IJCAI, AAAI, ICCV, and CVPR, and journal articles, such as IEEE Transactions on Pattern Analysis and Machine Intelligence, International Journal of Computer Vision, and Artificial Intelligence, which received more than 19000 citations from Google Scholar. His research interests include computer vision and computer gaming, with a current focus on human-computer interactive learning in complex decision-making scenarios.
Junliang Xing (Senior Member, IEEE) received the dual B.E. degree in computer science and applied mathematics from Xi’an Jiaotong University in 2007 and the Ph.D. degree in computer science and technology from Tsinghua University in 2012. He is currently a Professor with the Department of Computer Science and Technology, Tsinghua University. He has published over 120 peer-reviewed conference papers, such as IJCAI, AAAI, ICCV, and CVPR, and journal articles, such as IEEE Transactions on Pattern Analysis and Machine Intelligence, International Journal of Computer Vision, and Artificial Intelligence, which received more than 19000 citations from Google Scholar. His research interests include computer vision and computer gaming, with a current focus on human-computer interactive learning in complex decision-making scenarios.View more
Skywork AI, 2 Science park drive, Singapore
Shuicheng Yan (Fellow, IEEE) is currently the Director of Skywork AI, Singapore. He has authored or co-authored more than 600 papers in top international journals and conferences, with Google Scholar Citation more than 40000 times, and H-index of 105. His research interests include computer vision, machine learning, and multimedia analysis. He is a fellow of the Academy of Engineering, Singapore, an ACM Fellow, and an IAPR Fellow. He was among the Thomson Reuters Highly Cited Researchers in 2014, 2015, 2016, 2018, and 2019. His team has received winner or honorable-mention prizes for ten times of two core competitions, Pascal VOC and ImageNet (ILSVRC), which are deemed as World Cup in computer vision community. His team was a recipient of ten best paper or best student paper prizes and especially, a grand slam in ACM MM, the top conference in multimedia, including the Best Paper Award, the Best Student Paper Award, and the Best Demo Award.
Shuicheng Yan (Fellow, IEEE) is currently the Director of Skywork AI, Singapore. He has authored or co-authored more than 600 papers in top international journals and conferences, with Google Scholar Citation more than 40000 times, and H-index of 105. His research interests include computer vision, machine learning, and multimedia analysis. He is a fellow of the Academy of Engineering, Singapore, an ACM Fellow, and an IAPR Fellow. He was among the Thomson Reuters Highly Cited Researchers in 2014, 2015, 2016, 2018, and 2019. His team has received winner or honorable-mention prizes for ten times of two core competitions, Pascal VOC and ImageNet (ILSVRC), which are deemed as World Cup in computer vision community. His team was a recipient of ten best paper or best student paper prizes and especially, a grand slam in ACM MM, the top conference in multimedia, including the Best Paper Award, the Best Student Paper Award, and the Best Demo Award.View more
EVOL Lab, Institute of AI (TeleAI), China Telecom and the School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University (NWPU), Xi’an, China
Jian Zhao (Member, IEEE) received the Ph.D. degree from the National University of Singapore (NUS). He is currently the Leader of the EVOL Laboratory, a Principal Research Scientist with the Institute of AI (TeleAI), China Telecom, China, and a Researcher and the Ph.D. Supervisor of the School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University (NWPU), Xi’an, Shanxi, China. He has published over 60 CCF-A academic articles, including first-author IEEE Transactions on Pattern Analysis and Machine Intelligence✗2 (IF: 20.8) and International Journal of Computer Vision✗3 (IF: 11.6). He has also been authorized five national invention patents as the first inventor. The related technical achievements have been applied and verified in seven leading technology enterprises, including China Telecom, Baidu, Ant Financial, Qihoo, and 360, and have produced significant benefits. His research interests include vicinagearch security, AI + cultural tourism, and multi-modal AI agent. He served as a member for the board of directors of Beijing Society of Image and Graphics (BSIG) and the Editorial Board Member for the internationally renowned journals Pattern Recognition, Electronics and Signal Processing, Artificial Intelligence Advances, and IET Computer Vision. He is a Senior Member of China Society of Images and Graphics (CSIG) and Chinese Association for Artificial Intelligence (CAAI). He received the 2020–2022 Young Elite Scientist Sponsorship Program from China Association for Science and Technology (CAST) and the 2021–2023 Beijing Young Elite Scientist Sponsorship Program from Beijing Association for Science and Technology (BAST). He is in charge of seven relevant projects supported by the National Nature Science Foundation of China (NSFC). He has won the WU WEN JUN AI Outstanding Youth Award (2023), the First-Prize of the WU WEN JUN AI Natural Science Award (2/5, 2022), the PREMIA Lee Hwee Kuan Award (2019), and the ACM Multimedia Best Student Paper Award (first author, 1/208, CCF-A conference, 2018). He has also won eight winner awards in domestic and foreign technical challenges. He also served as the Senior Area Chair for the Vision and Learning Seminar (VALSE), the Session Chair for the ACM Multimedia 2021, the Area Chair for the CICAI 2022/2023, and the Workshop Chair for the CCBR 2024. He served as the Guest Editor for Pattern Recognition Letters Special Issue on Recent Advances in Deep Learning Model Security and Electronics Special Issue on Multimedia Content Analysis, Management and Retrieval: Trends and Challenges.
Jian Zhao (Member, IEEE) received the Ph.D. degree from the National University of Singapore (NUS). He is currently the Leader of the EVOL Laboratory, a Principal Research Scientist with the Institute of AI (TeleAI), China Telecom, China, and a Researcher and the Ph.D. Supervisor of the School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University (NWPU), Xi’an, Shanxi, China. He has published over 60 CCF-A academic articles, including first-author IEEE Transactions on Pattern Analysis and Machine Intelligence✗2 (IF: 20.8) and International Journal of Computer Vision✗3 (IF: 11.6). He has also been authorized five national invention patents as the first inventor. The related technical achievements have been applied and verified in seven leading technology enterprises, including China Telecom, Baidu, Ant Financial, Qihoo, and 360, and have produced significant benefits. His research interests include vicinagearch security, AI + cultural tourism, and multi-modal AI agent. He served as a member for the board of directors of Beijing Society of Image and Graphics (BSIG) and the Editorial Board Member for the internationally renowned journals Pattern Recognition, Electronics and Signal Processing, Artificial Intelligence Advances, and IET Computer Vision. He is a Senior Member of China Society of Images and Graphics (CSIG) and Chinese Association for Artificial Intelligence (CAAI). He received the 2020–2022 Young Elite Scientist Sponsorship Program from China Association for Science and Technology (CAST) and the 2021–2023 Beijing Young Elite Scientist Sponsorship Program from Beijing Association for Science and Technology (BAST). He is in charge of seven relevant projects supported by the National Nature Science Foundation of China (NSFC). He has won the WU WEN JUN AI Outstanding Youth Award (2023), the First-Prize of the WU WEN JUN AI Natural Science Award (2/5, 2022), the PREMIA Lee Hwee Kuan Award (2019), and the ACM Multimedia Best Student Paper Award (first author, 1/208, CCF-A conference, 2018). He has also won eight winner awards in domestic and foreign technical challenges. He also served as the Senior Area Chair for the Vision and Learning Seminar (VALSE), the Session Chair for the ACM Multimedia 2021, the Area Chair for the CICAI 2022/2023, and the Workshop Chair for the CCBR 2024. He served as the Guest Editor for Pattern Recognition Letters Special Issue on Recent Advances in Deep Learning Model Security and Electronics Special Issue on Multimedia Content Analysis, Management and Retrieval: Trends and Challenges.View more
