Abstract:
Vision-based sign language translation (SLT) is a challenging task due to the complicated variations of facial expressions, gestures, and articulated poses involved in si...Show MoreMetadata
Abstract:
Vision-based sign language translation (SLT) is a challenging task due to the complicated variations of facial expressions, gestures, and articulated poses involved in sign linguistics. As a weakly supervised sequence-to-sequence learning problem, in SLT there are usually no exact temporal boundaries of actions. To adequately explore temporal hints in videos, we propose a novel framework named Hierarchical deep Recurrent Fusion (HRF). Aiming at modeling discriminative action patterns, in HRF we design an adaptive temporal encoder to capture crucial RGB visemes and skeleton signees. Specifically, RGB visemes and skeleton signees are learned by the same scheme named Adaptive Clip Summarization (ACS), respectively. ACS consists of three key modules, i.e., variable-length clip mining, adaptive temporal pooling, and attention-aware weighting. Besides, based on unaligned action patterns (RGB visemes and skeleton signees), a query-adaptive decoding fusion is proposed to translate the target sentence. Extensive experiments demonstrate the effectiveness of the proposed HRF framework.
Published in: IEEE Transactions on Image Processing ( Volume: 29)
Funding Agency:
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China
Dan Guo received the B.E. degree in computer science and technology from Yangtze University, China, in 2004, and the Ph.D. degree in system analysis and integration from the Huazhong University of Science and Technology, China, in 2010. She is currently an Associate Professor with the School of Computer Science and Information Engineering, Hefei University of Technology, China. Her research interests include computer visi...Show More
Dan Guo received the B.E. degree in computer science and technology from Yangtze University, China, in 2004, and the Ph.D. degree in system analysis and integration from the Huazhong University of Science and Technology, China, in 2010. She is currently an Associate Professor with the School of Computer Science and Information Engineering, Hefei University of Technology, China. Her research interests include computer visi...View more
EEIS Department, University of Science and Technology of China, Hefei, China
Wengang Zhou received the B.E. degree in electronic information engineering from Wuhan University, China, in 2006, and the Ph.D. degree in electronic engineering and information science from the University of Science and Technology of China (USTC), China, in 2011. From September 2011 to 2013, he was a Postdoctoral Researcher with the Computer Science Department, The University of Texas at San Antonio. He is currently a Pr...Show More
Wengang Zhou received the B.E. degree in electronic information engineering from Wuhan University, China, in 2006, and the Ph.D. degree in electronic engineering and information science from the University of Science and Technology of China (USTC), China, in 2011. From September 2011 to 2013, he was a Postdoctoral Researcher with the Computer Science Department, The University of Texas at San Antonio. He is currently a Pr...View more
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China
Huawei Cloud AI Platform, Jiangsu, China
Anyang Li received the B.E. degree in computer science and technology from Nanjing Normal University, China, in 2016, and the M.S. degree in computer technology from the Hefei University of Technology, China, in 2019. He is currently a Software Development Engineer with Huawei Cloud AI Platform. His research interests include computer vision, big data analysis, and distributed computing.
Anyang Li received the B.E. degree in computer science and technology from Nanjing Normal University, China, in 2016, and the M.S. degree in computer technology from the Hefei University of Technology, China, in 2019. He is currently a Software Development Engineer with Huawei Cloud AI Platform. His research interests include computer vision, big data analysis, and distributed computing.View more
EEIS Department, University of Science and Technology of China, Hefei, China
Houqiang Li (M’10–SM’12) received the B.S., M.Eng., and Ph.D. degrees in electronic engineering from the University of Science and Technology of China (USTC), Hefei, China, in 1992, 1997, and 2000, respectively. He is currently a Professor with the Department of Electronic Engineering and Information Science, USTC. He has authored or coauthored over 100 articles in journals and conferences. His research interests include ...Show More
Houqiang Li (M’10–SM’12) received the B.S., M.Eng., and Ph.D. degrees in electronic engineering from the University of Science and Technology of China (USTC), Hefei, China, in 1992, 1997, and 2000, respectively. He is currently a Professor with the Department of Electronic Engineering and Information Science, USTC. He has authored or coauthored over 100 articles in journals and conferences. His research interests include ...View more
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China
Meng Wang (SM’17) received the B.E. and Ph.D. degrees, in the special class for the gifted young, from the Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China, in 2003 and 2008, respectively. He is currently a Professor with the Hefei University of Technology, China. His current research interests include multimedia content analysis, computer vision, an...Show More
Meng Wang (SM’17) received the B.E. and Ph.D. degrees, in the special class for the gifted young, from the Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China, in 2003 and 2008, respectively. He is currently a Professor with the Hefei University of Technology, China. His current research interests include multimedia content analysis, computer vision, an...View more
Contents School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China
Dan Guo received the B.E. degree in computer science and technology from Yangtze University, China, in 2004, and the Ph.D. degree in system analysis and integration from the Huazhong University of Science and Technology, China, in 2010. She is currently an Associate Professor with the School of Computer Science and Information Engineering, Hefei University of Technology, China. Her research interests include computer vision, machine learning, and intelligent multimedia content analysis.
Dan Guo received the B.E. degree in computer science and technology from Yangtze University, China, in 2004, and the Ph.D. degree in system analysis and integration from the Huazhong University of Science and Technology, China, in 2010. She is currently an Associate Professor with the School of Computer Science and Information Engineering, Hefei University of Technology, China. Her research interests include computer vision, machine learning, and intelligent multimedia content analysis.View more
EEIS Department, University of Science and Technology of China, Hefei, China
Wengang Zhou received the B.E. degree in electronic information engineering from Wuhan University, China, in 2006, and the Ph.D. degree in electronic engineering and information science from the University of Science and Technology of China (USTC), China, in 2011. From September 2011 to 2013, he was a Postdoctoral Researcher with the Computer Science Department, The University of Texas at San Antonio. He is currently a Professor with the EEIS Department, USTC. His research interests include multimedia information retrieval and computer vision.
Wengang Zhou received the B.E. degree in electronic information engineering from Wuhan University, China, in 2006, and the Ph.D. degree in electronic engineering and information science from the University of Science and Technology of China (USTC), China, in 2011. From September 2011 to 2013, he was a Postdoctoral Researcher with the Computer Science Department, The University of Texas at San Antonio. He is currently a Professor with the EEIS Department, USTC. His research interests include multimedia information retrieval and computer vision.View more
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China
Huawei Cloud AI Platform, Jiangsu, China
Anyang Li received the B.E. degree in computer science and technology from Nanjing Normal University, China, in 2016, and the M.S. degree in computer technology from the Hefei University of Technology, China, in 2019. He is currently a Software Development Engineer with Huawei Cloud AI Platform. His research interests include computer vision, big data analysis, and distributed computing.
Anyang Li received the B.E. degree in computer science and technology from Nanjing Normal University, China, in 2016, and the M.S. degree in computer technology from the Hefei University of Technology, China, in 2019. He is currently a Software Development Engineer with Huawei Cloud AI Platform. His research interests include computer vision, big data analysis, and distributed computing.View more
EEIS Department, University of Science and Technology of China, Hefei, China
Houqiang Li (M’10–SM’12) received the B.S., M.Eng., and Ph.D. degrees in electronic engineering from the University of Science and Technology of China (USTC), Hefei, China, in 1992, 1997, and 2000, respectively. He is currently a Professor with the Department of Electronic Engineering and Information Science, USTC. He has authored or coauthored over 100 articles in journals and conferences. His research interests include multimedia search, image/video analysis, and video coding and communication. He was a recipient of the Best Paper Award at the Visual Communications and Image Processing in 2012, the Best Paper Award at the International Conference on Internet Multimedia Computing and Service in 2012, and the Best Paper Award at the International Conference on Mobile and Ubiquitous Multimedia from ACM in 2011. He was a Senior Author of the Best Student Paper of the 5th International Mobile Multimedia Communications Conference in 2009. He has served on technical/program committees and organizing committees and as the program co-chair or the track/session chair for over ten international conferences. He has served an Associate Editor for the IEEE Transactions on Circuit and Systems for Video Technology from 2010 to 2013. He has been serving on the Editorial Board for the Journal of Multimedia since 2009.
Houqiang Li (M’10–SM’12) received the B.S., M.Eng., and Ph.D. degrees in electronic engineering from the University of Science and Technology of China (USTC), Hefei, China, in 1992, 1997, and 2000, respectively. He is currently a Professor with the Department of Electronic Engineering and Information Science, USTC. He has authored or coauthored over 100 articles in journals and conferences. His research interests include multimedia search, image/video analysis, and video coding and communication. He was a recipient of the Best Paper Award at the Visual Communications and Image Processing in 2012, the Best Paper Award at the International Conference on Internet Multimedia Computing and Service in 2012, and the Best Paper Award at the International Conference on Mobile and Ubiquitous Multimedia from ACM in 2011. He was a Senior Author of the Best Student Paper of the 5th International Mobile Multimedia Communications Conference in 2009. He has served on technical/program committees and organizing committees and as the program co-chair or the track/session chair for over ten international conferences. He has served an Associate Editor for the IEEE Transactions on Circuit and Systems for Video Technology from 2010 to 2013. He has been serving on the Editorial Board for the Journal of Multimedia since 2009.View more
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China
Meng Wang (SM’17) received the B.E. and Ph.D. degrees, in the special class for the gifted young, from the Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China, in 2003 and 2008, respectively. He is currently a Professor with the Hefei University of Technology, China. His current research interests include multimedia content analysis, computer vision, and pattern recognition. He has authored over 200 book chapters and journal and conference articles in these areas. He was a recipient of the ACM SIGMM Rising Star Award in 2014. He is also an Associate Editor of the IEEE Transactions on Knowledge and Data Engineering, the IEEE Transactions on Circuits and Systems for Video Technology, and the IEEE Transactions on Neural Networks and Learning Systems.
Meng Wang (SM’17) received the B.E. and Ph.D. degrees, in the special class for the gifted young, from the Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China, in 2003 and 2008, respectively. He is currently a Professor with the Hefei University of Technology, China. His current research interests include multimedia content analysis, computer vision, and pattern recognition. He has authored over 200 book chapters and journal and conference articles in these areas. He was a recipient of the ACM SIGMM Rising Star Award in 2014. He is also an Associate Editor of the IEEE Transactions on Knowledge and Data Engineering, the IEEE Transactions on Circuits and Systems for Video Technology, and the IEEE Transactions on Neural Networks and Learning Systems.View more
