Abstract:
Despite the recent success of deep learning in video-related tasks, deep models typically focus on the most discriminative features, ignoring other potentially non-trivia...Show MoreMetadata
Abstract:
Despite the recent success of deep learning in video-related tasks, deep models typically focus on the most discriminative features, ignoring other potentially non-trivial and informative contents. Such characteristic heavily constrains their capability to learn implicit visual grammars in sign videos behind the collaboration of different visual cues (i.e., hand shape, facial expression and body posture). To this end, we approach video-based sign language understanding with multi-cue learning and propose a spatial-temporal multi-cue (STMC) network to solve the vision-based sequence learning problem. Our STMC network consists of a spatial multi-cue (SMC) module and a temporal multi-cue (TMC) module. The SMC module learns to spatial representation of different cues with a self-contained pose estimation branch. The TMC module models temporal corrections from intra-cue and inter-cue perspectives to explore the collaboration of multiple cues. A joint optimization strategy and a segmented attention mechanism are designed to make the best of multi-cue sources for SL recognition and translation. To validate the effectiveness, we perform experiments on three large-scale sign language benchmarks: PHOENIX-2014, CSL and PHOENIX-2014-T. Experimental results demonstrate that the proposed method achieves new state-of-the-art performance on all three benchmarks.
Published in: IEEE Transactions on Multimedia ( Volume: 24)
Funding Agency:
CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System, Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China
Hao Zhou received the B.S. degree in communication engineering from Xidian University, Xi’an, China, in 2017. He is currently working toward the Ph.D. degree with the Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China. His research interests include computer vision and sign language processing.
Hao Zhou received the B.S. degree in communication engineering from Xidian University, Xi’an, China, in 2017. He is currently working toward the Ph.D. degree with the Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China. His research interests include computer vision and sign language processing.View more
CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System, Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China
Wengang Zhou received the B.E. degree in electronic information engineering from Wuhan University, Hubei, China, and the Ph.D. degree in electronic engineering and information science from the University of Science and Technology of China (USTC), Hefei, China, in 2006 and 2011, respectively. From 2011 to 2013, he was a Postdoc Researcher with Computer Science Department, University of Texas at San Antonio, San Antonio, TX...Show More
Wengang Zhou received the B.E. degree in electronic information engineering from Wuhan University, Hubei, China, and the Ph.D. degree in electronic engineering and information science from the University of Science and Technology of China (USTC), Hefei, China, in 2006 and 2011, respectively. From 2011 to 2013, he was a Postdoc Researcher with Computer Science Department, University of Texas at San Antonio, San Antonio, TX...View more
CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System, Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China
Yun Zhou received the B.S. degree in electronic and information engineering from Anhui Normal University, Wuhu, China, in 2010, and the Ph.D. degree in communication and information system from the Hefei University of Technology, Hefei, China, in 2018. She is currently a Postdoctoral Researcher with the Electronic Engineering and Information Science Department, University of Science and Technology of China, Hefei, China. ...Show More
Yun Zhou received the B.S. degree in electronic and information engineering from Anhui Normal University, Wuhu, China, in 2010, and the Ph.D. degree in communication and information system from the Hefei University of Technology, Hefei, China, in 2018. She is currently a Postdoctoral Researcher with the Electronic Engineering and Information Science Department, University of Science and Technology of China, Hefei, China. ...View more
CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System, Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China
Houqiang Li (Fellow, IEEE) received the B.S., M.Eng., and Ph.D. degrees in electronic engineering from the University of Science and Technology of China, Hefei, China, in 1992, 1997, and 2000, respectively. He is currently a Professor with the Department of Electronic Engineering and Information Science, University of Science and Technology of China. He has authored and coauthored more than 200 papers in journals and conf...Show More
Houqiang Li (Fellow, IEEE) received the B.S., M.Eng., and Ph.D. degrees in electronic engineering from the University of Science and Technology of China, Hefei, China, in 1992, 1997, and 2000, respectively. He is currently a Professor with the Department of Electronic Engineering and Information Science, University of Science and Technology of China. He has authored and coauthored more than 200 papers in journals and conf...View more
Contents CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System, Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China
Hao Zhou received the B.S. degree in communication engineering from Xidian University, Xi’an, China, in 2017. He is currently working toward the Ph.D. degree with the Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China. His research interests include computer vision and sign language processing.
Hao Zhou received the B.S. degree in communication engineering from Xidian University, Xi’an, China, in 2017. He is currently working toward the Ph.D. degree with the Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China. His research interests include computer vision and sign language processing.View more
CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System, Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China
Wengang Zhou received the B.E. degree in electronic information engineering from Wuhan University, Hubei, China, and the Ph.D. degree in electronic engineering and information science from the University of Science and Technology of China (USTC), Hefei, China, in 2006 and 2011, respectively. From 2011 to 2013, he was a Postdoc Researcher with Computer Science Department, University of Texas at San Antonio, San Antonio, TX, USA. He is currently a Professor with the Department of Electrical Engineering and Information Systems, USTC.
His research interests include multimedia information retrieval and computer vision.
Wengang Zhou received the B.E. degree in electronic information engineering from Wuhan University, Hubei, China, and the Ph.D. degree in electronic engineering and information science from the University of Science and Technology of China (USTC), Hefei, China, in 2006 and 2011, respectively. From 2011 to 2013, he was a Postdoc Researcher with Computer Science Department, University of Texas at San Antonio, San Antonio, TX, USA. He is currently a Professor with the Department of Electrical Engineering and Information Systems, USTC.
His research interests include multimedia information retrieval and computer vision.View more
CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System, Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China
Yun Zhou received the B.S. degree in electronic and information engineering from Anhui Normal University, Wuhu, China, in 2010, and the Ph.D. degree in communication and information system from the Hefei University of Technology, Hefei, China, in 2018. She is currently a Postdoctoral Researcher with the Electronic Engineering and Information Science Department, University of Science and Technology of China, Hefei, China. Her research interests include computer vision and reinforcement learning.
Yun Zhou received the B.S. degree in electronic and information engineering from Anhui Normal University, Wuhu, China, in 2010, and the Ph.D. degree in communication and information system from the Hefei University of Technology, Hefei, China, in 2018. She is currently a Postdoctoral Researcher with the Electronic Engineering and Information Science Department, University of Science and Technology of China, Hefei, China. Her research interests include computer vision and reinforcement learning.View more
CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System, Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China
Houqiang Li (Fellow, IEEE) received the B.S., M.Eng., and Ph.D. degrees in electronic engineering from the University of Science and Technology of China, Hefei, China, in 1992, 1997, and 2000, respectively. He is currently a Professor with the Department of Electronic Engineering and Information Science, University of Science and Technology of China. He has authored and coauthored more than 200 papers in journals and conferences. His research interests include multimedia search, image/video analysis, video coding, and communication. From 2010 to 2013, he was an Associate Editor for the IEEE Transactions On Circuits And Systems For Video Technology. He was the TPC Co-Chair of VCIP 2010 and is the General Co-Chair of ICME 2021. He was the recipient of the National Science Funds (NSFC) for Distinguished Young Scientists, the Distinguished Professor of Changjiang Scholars Program of China, the Leading Scientist of Ten Thousand Talent Program of China, the National Technological Invention Award of China (second class) in 2019, the National Natural Science Award of China (second class) in 2015, the Best Paper Award for VCIP 2012, the Best Paper Award for ICIMCS 2012, and the Best Paper Award for ACM MUM in 2011.
Houqiang Li (Fellow, IEEE) received the B.S., M.Eng., and Ph.D. degrees in electronic engineering from the University of Science and Technology of China, Hefei, China, in 1992, 1997, and 2000, respectively. He is currently a Professor with the Department of Electronic Engineering and Information Science, University of Science and Technology of China. He has authored and coauthored more than 200 papers in journals and conferences. His research interests include multimedia search, image/video analysis, video coding, and communication. From 2010 to 2013, he was an Associate Editor for the IEEE Transactions On Circuits And Systems For Video Technology. He was the TPC Co-Chair of VCIP 2010 and is the General Co-Chair of ICME 2021. He was the recipient of the National Science Funds (NSFC) for Distinguished Young Scientists, the Distinguished Professor of Changjiang Scholars Program of China, the Leading Scientist of Ten Thousand Talent Program of China, the National Technological Invention Award of China (second class) in 2019, the National Natural Science Award of China (second class) in 2015, the Best Paper Award for VCIP 2012, the Best Paper Award for ICIMCS 2012, and the Best Paper Award for ACM MUM in 2011.View more
