Abstract:
Sign language translation (SLT) is a challenging weakly supervised task without word-level annotations. An effective method of SLT is to leverage multimodal complementari...Show MoreMetadata
Abstract:
Sign language translation (SLT) is a challenging weakly supervised task without word-level annotations. An effective method of SLT is to leverage multimodal complementarity and to explore implicit temporal cues. In this work, we propose a graph-based multimodal sequential embedding network (MSeqGraph), in which multiple sequential modalities are densely correlated. Specifically, we build a graph structure to realize the intra-modal and inter-modal correlations. First, we design a graph embedding unit (GEU), which embeds a parallel convolution with channel-wise and temporal-wise learning into the graph convolution to learn the temporal cues in each modal sequence and cross-modal complementarity. Then, a hierarchical GEU stacker with a pooling-based skip connection is proposed. Unlike the state-of-the-art methods, to obtain a compact and informative representation of multimodal sequences, the GEU stacker gradually compresses the channel d with multi-modalities m rather than the temporal dimension t. Finally, we adopt the connectionist temporal decoding strategy to explore the entire video’s temporal transition and translate the sentence. Extensive experiments on the USTC-CSL and BOSTON-104 datasets demonstrate the effectiveness of the proposed method.
Published in: IEEE Transactions on Multimedia ( Volume: 24)
Funding Agency:

Key Laboratory of Knowledge Engineering with Big Data (HFUT), Ministry of Education, Hefei, China
Intelligent Interconnected Systems Laboratory of Anhui Province (HFUT), Hefei, China
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China
Shengeng Tang received the B.E. degree in computer science and technology from Hunan Normal University, China, in 2017. He is currently working toward the Ph.D. degree with the School of Computer Science and Information Engineering, Hefei University of Technology, China. His research interests include multimedia content analysis and computer vision.
Shengeng Tang received the B.E. degree in computer science and technology from Hunan Normal University, China, in 2017. He is currently working toward the Ph.D. degree with the School of Computer Science and Information Engineering, Hefei University of Technology, China. His research interests include multimedia content analysis and computer vision.View more

Key Laboratory of Knowledge Engineering with Big Data (HFUT), Ministry of Education, Hefei, China
Intelligent Interconnected Systems Laboratory of Anhui Province (HFUT), Hefei, China
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China
Dan Guo received the B.E. degree in computer science and technology from Yangtze University, China, in 2004, and the Ph.D. degree in system analysis and integration from the Huazhong University of Science and Technology, China, in 2010. She is currently a Professor with the School of Computer Science and Information Engineering, Hefei University of Technology, China. Her research interests include computer vision, machine...Show More
Dan Guo received the B.E. degree in computer science and technology from Yangtze University, China, in 2004, and the Ph.D. degree in system analysis and integration from the Huazhong University of Science and Technology, China, in 2010. She is currently a Professor with the School of Computer Science and Information Engineering, Hefei University of Technology, China. Her research interests include computer vision, machine...View more

Key Laboratory of Knowledge Engineering with Big Data (HFUT), Ministry of Education, Hefei, China
Intelligent Interconnected Systems Laboratory of Anhui Province (HFUT), Hefei, China
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China
Richang Hong received the Ph.D. degree from the University of Science and Technology of China, Hefei, China, in 2008. He is currently a Professor with the Hefei University of Technology, Hefei. His research interests include multimedia content analysis and social media, in which he has coauthored more than 100 publications. He is a member of the ACM and an Executive Committee Member of the ACM SIGMM China Chapter. He was ...Show More
Richang Hong received the Ph.D. degree from the University of Science and Technology of China, Hefei, China, in 2008. He is currently a Professor with the Hefei University of Technology, Hefei. His research interests include multimedia content analysis and social media, in which he has coauthored more than 100 publications. He is a member of the ACM and an Executive Committee Member of the ACM SIGMM China Chapter. He was ...View more

Key Laboratory of Knowledge Engineering with Big Data (HFUT), Ministry of Education, Hefei, China
Intelligent Interconnected Systems Laboratory of Anhui Province (HFUT), Hefei, China
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China
Meng Wang (Fellow, IEEE) received the B.E. degree and Ph.D. degree from the Special Class for the Gifted Young and the Department of Electronic Engineering and Information Science from the University of Science and Technology of China (USTC), Hefei, China, in 2003 and 2008, respectively. He is a Professor with the Hefei University of Technology, China. He has authored more than 200 book chapters, journal and conference pa...Show More
Meng Wang (Fellow, IEEE) received the B.E. degree and Ph.D. degree from the Special Class for the Gifted Young and the Department of Electronic Engineering and Information Science from the University of Science and Technology of China (USTC), Hefei, China, in 2003 and 2008, respectively. He is a Professor with the Hefei University of Technology, China. He has authored more than 200 book chapters, journal and conference pa...View more

Key Laboratory of Knowledge Engineering with Big Data (HFUT), Ministry of Education, Hefei, China
Intelligent Interconnected Systems Laboratory of Anhui Province (HFUT), Hefei, China
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China
Shengeng Tang received the B.E. degree in computer science and technology from Hunan Normal University, China, in 2017. He is currently working toward the Ph.D. degree with the School of Computer Science and Information Engineering, Hefei University of Technology, China. His research interests include multimedia content analysis and computer vision.
Shengeng Tang received the B.E. degree in computer science and technology from Hunan Normal University, China, in 2017. He is currently working toward the Ph.D. degree with the School of Computer Science and Information Engineering, Hefei University of Technology, China. His research interests include multimedia content analysis and computer vision.View more

Key Laboratory of Knowledge Engineering with Big Data (HFUT), Ministry of Education, Hefei, China
Intelligent Interconnected Systems Laboratory of Anhui Province (HFUT), Hefei, China
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China
Dan Guo received the B.E. degree in computer science and technology from Yangtze University, China, in 2004, and the Ph.D. degree in system analysis and integration from the Huazhong University of Science and Technology, China, in 2010. She is currently a Professor with the School of Computer Science and Information Engineering, Hefei University of Technology, China. Her research interests include computer vision, machine learning, and intelligent multimedia content analysis.
Dan Guo received the B.E. degree in computer science and technology from Yangtze University, China, in 2004, and the Ph.D. degree in system analysis and integration from the Huazhong University of Science and Technology, China, in 2010. She is currently a Professor with the School of Computer Science and Information Engineering, Hefei University of Technology, China. Her research interests include computer vision, machine learning, and intelligent multimedia content analysis.View more

Key Laboratory of Knowledge Engineering with Big Data (HFUT), Ministry of Education, Hefei, China
Intelligent Interconnected Systems Laboratory of Anhui Province (HFUT), Hefei, China
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China
Richang Hong received the Ph.D. degree from the University of Science and Technology of China, Hefei, China, in 2008. He is currently a Professor with the Hefei University of Technology, Hefei. His research interests include multimedia content analysis and social media, in which he has coauthored more than 100 publications. He is a member of the ACM and an Executive Committee Member of the ACM SIGMM China Chapter. He was the recipient of the Best Paper Award at the ACM Multimedia 2010, the Best Paper Award at the ACM ICMR 2015, and the Honorable Mention of the IEEE Transactions on Multimedia Best Paper Award 2015. He was an Associate Editor for the IEEE Multimedia Magazine, Information Sciences and Signal Processing, Elsevier, and was the Technical Program Chair of the MMM 2016, ICIMCS 2017, and PCM 2018.
Richang Hong received the Ph.D. degree from the University of Science and Technology of China, Hefei, China, in 2008. He is currently a Professor with the Hefei University of Technology, Hefei. His research interests include multimedia content analysis and social media, in which he has coauthored more than 100 publications. He is a member of the ACM and an Executive Committee Member of the ACM SIGMM China Chapter. He was the recipient of the Best Paper Award at the ACM Multimedia 2010, the Best Paper Award at the ACM ICMR 2015, and the Honorable Mention of the IEEE Transactions on Multimedia Best Paper Award 2015. He was an Associate Editor for the IEEE Multimedia Magazine, Information Sciences and Signal Processing, Elsevier, and was the Technical Program Chair of the MMM 2016, ICIMCS 2017, and PCM 2018.View more

Key Laboratory of Knowledge Engineering with Big Data (HFUT), Ministry of Education, Hefei, China
Intelligent Interconnected Systems Laboratory of Anhui Province (HFUT), Hefei, China
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China
Meng Wang (Fellow, IEEE) received the B.E. degree and Ph.D. degree from the Special Class for the Gifted Young and the Department of Electronic Engineering and Information Science from the University of Science and Technology of China (USTC), Hefei, China, in 2003 and 2008, respectively. He is a Professor with the Hefei University of Technology, China. He has authored more than 200 book chapters, journal and conference papers in these areas. His current research interests include multimedia content analysis, computer vision, and pattern recognition. He was the recipient of the ACM SIGMM Rising Star Award 2014. He is an Associate Editor for IEEE Transactions on Knowledge and Data Engineering(IEEE TKDE),IEEE Transactions on Circuits and Systems for Video Technology (IEEE TCSVT), IEEE Transactions on Multimedia (IEEE TMM), and IEEE Transactions on Neural Networks and Learning Systems (IEEE TNNLS).
Meng Wang (Fellow, IEEE) received the B.E. degree and Ph.D. degree from the Special Class for the Gifted Young and the Department of Electronic Engineering and Information Science from the University of Science and Technology of China (USTC), Hefei, China, in 2003 and 2008, respectively. He is a Professor with the Hefei University of Technology, China. He has authored more than 200 book chapters, journal and conference papers in these areas. His current research interests include multimedia content analysis, computer vision, and pattern recognition. He was the recipient of the ACM SIGMM Rising Star Award 2014. He is an Associate Editor for IEEE Transactions on Knowledge and Data Engineering(IEEE TKDE),IEEE Transactions on Circuits and Systems for Video Technology (IEEE TCSVT), IEEE Transactions on Multimedia (IEEE TMM), and IEEE Transactions on Neural Networks and Learning Systems (IEEE TNNLS).View more