Loading [MathJax]/extensions/MathMenu.js
HAKE: A Knowledge Engine Foundation for Human Activity Understanding | IEEE Journals & Magazine | IEEE Xplore

HAKE: A Knowledge Engine Foundation for Human Activity Understanding


Abstract:

Human activity understanding is of widespread interest in artificial intelligence and spans diverse applications like health care and behavior analysis. Although there ha...Show More

Abstract:

Human activity understanding is of widespread interest in artificial intelligence and spans diverse applications like health care and behavior analysis. Although there have been advances with deep learning, it remains challenging. The object recognition-like solutions usually try to map pixels to semantics directly, but activity patterns are much different from object patterns, thus hindering another success. In this article, we propose a novel paradigm to reformulate this task in two-stage: first mapping pixels to an intermediate space spanned by atomic activity primitives, then programming detected primitives with interpretable logic rules to infer semantics. To afford a representative primitive space, we build a knowledge base including 26+ M primitive labels and logic rules from human priors or automatic discovering. Our framework, Human Activity Knowledge Engine (HAKE), exhibits superior generalization ability and performance upon canonical methods on challenging benchmarks. Code and data are available at http://hake-mvig.cn/.
Page(s): 8494 - 8506
Date of Publication: 29 December 2022

ISSN Information:

PubMed ID: 37819797

Funding Agency:

Author image of Yong-Lu Li
Qing Yuan Research Institute, Shanghai Jiao Tong University, Shanghai, China
Yong-Lu Li received the PhD degree from Shanghai Jiao Tong University. He is a tenure-track Assistant Professor with Shanghai Jiao Tong University. Before joining SJTU, he was a postdoc fellow with HKUST and worked closely with Prof. CK Tang and Yu-Wing Tai. He was awarded the Baidu Scholarship, YunFan award, Shanghai Outstanding Graduate, etc. His research interests include computer vision, reasoning, and embodied AI.
Yong-Lu Li received the PhD degree from Shanghai Jiao Tong University. He is a tenure-track Assistant Professor with Shanghai Jiao Tong University. Before joining SJTU, he was a postdoc fellow with HKUST and worked closely with Prof. CK Tang and Yu-Wing Tai. He was awarded the Baidu Scholarship, YunFan award, Shanghai Outstanding Graduate, etc. His research interests include computer vision, reasoning, and embodied AI.View more
Author image of Xinpeng Liu
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Xinpeng Liu received the BE degree from Shanghai Jiao Tong University, Shanghai, China, where he is currently working toward the master's degree in computer science. His research interests include computer vision and deep learning.
Xinpeng Liu received the BE degree from Shanghai Jiao Tong University, Shanghai, China, where he is currently working toward the master's degree in computer science. His research interests include computer vision and deep learning.View more
Author image of Xiaoqian Wu
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Xiaoqian Wu is currently working toward the bachelor's degree with Shanghai Jiao Tong University, Shanghai, China. She is a research assistant with Machine Vision and Intelligence Group, Shanghai Jiao Tong University. Her research interests include computer vision and deep learning.
Xiaoqian Wu is currently working toward the bachelor's degree with Shanghai Jiao Tong University, Shanghai, China. She is a research assistant with Machine Vision and Intelligence Group, Shanghai Jiao Tong University. Her research interests include computer vision and deep learning.View more
Author image of Yizhuo Li
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Yizhuo Li received the BS degree in electronic engineering from Tsinghua University, Beijing, China, in 2019. He is currently working toward the master degree with Shanghai Jiao Tong University, supervised by Prof. Cewu Lu. His research interests mainly focus on computer vision.
Yizhuo Li received the BS degree in electronic engineering from Tsinghua University, Beijing, China, in 2019. He is currently working toward the master degree with Shanghai Jiao Tong University, supervised by Prof. Cewu Lu. His research interests mainly focus on computer vision.View more
Author image of Zuoyu Qiu
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Zuoyu Qiu is currently working toward the bachelor's degree with Shanghai Jiao Tong University, Shanghai, China. He is a research assistant with Machine Vision and Intelligence Group, Shanghai Jiao Tong University. His research interests include computer vision and deep learning.
Zuoyu Qiu is currently working toward the bachelor's degree with Shanghai Jiao Tong University, Shanghai, China. He is a research assistant with Machine Vision and Intelligence Group, Shanghai Jiao Tong University. His research interests include computer vision and deep learning.View more
Author image of Liang Xu
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Liang Xu received the master's degree in computer science from Shanghai Jiao Tong University, supervised by Prof. Cewu Lu. His research interests mainly focus on computer vision.
Liang Xu received the master's degree in computer science from Shanghai Jiao Tong University, supervised by Prof. Cewu Lu. His research interests mainly focus on computer vision.View more
Author image of Yue Xu
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Yue Xu received the BE degree from Shanghai Jiao Tong University, Shanghai, China, where he is currently working toward the master's degree in computer science. His research interests mainly include computer vision.
Yue Xu received the BE degree from Shanghai Jiao Tong University, Shanghai, China, where he is currently working toward the master's degree in computer science. His research interests mainly include computer vision.View more
Author image of Hao-Shu Fang
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Hao-Shu Fang received the BE degree in computer science from Shanghai Jiao Tong University, in 2019. He is currently working toward the PhD degree with the Department of Computer Science and Engineering, Shanghai Jiao Tong University. His research interests include computer vision and robot intelligence. He was awarded the CCF-CV Rising Scholar Award in 2019, Baidu Fellowship in 2019, MSRA Fellowship in 2020, and ByteDanc...Show More
Hao-Shu Fang received the BE degree in computer science from Shanghai Jiao Tong University, in 2019. He is currently working toward the PhD degree with the Department of Computer Science and Engineering, Shanghai Jiao Tong University. His research interests include computer vision and robot intelligence. He was awarded the CCF-CV Rising Scholar Award in 2019, Baidu Fellowship in 2019, MSRA Fellowship in 2020, and ByteDanc...View more
Author image of Cewu Lu
Shanghai Digital Medicine Innovation Center, Ruijin Hospital, Shanghai, China
MoE Key Lab of AI, AI Institute, Shanghai Jiao Tong University, Shanghai, China
Shanghai Qi Zhi Institute, Shanghai, China
Cewu Lu received the PhD degree from the Chinese University of Hong Kong, supervised by Prof. Jiaya Jia. He is a professor with Shanghai Jiao Tong University. Before joining SJTU, he was a research fellow with Stanford University, working under Prof. Fei-Fei Li and Prof. Leonidas J. Guibas. He was a research assistant professor with the Hong Kong University of Science and Technology with Prof. Chi Keung Tang. His research...Show More
Cewu Lu received the PhD degree from the Chinese University of Hong Kong, supervised by Prof. Jiaya Jia. He is a professor with Shanghai Jiao Tong University. Before joining SJTU, he was a research fellow with Stanford University, working under Prof. Fei-Fei Li and Prof. Leonidas J. Guibas. He was a research assistant professor with the Hong Kong University of Science and Technology with Prof. Chi Keung Tang. His research...View more

Author image of Yong-Lu Li
Qing Yuan Research Institute, Shanghai Jiao Tong University, Shanghai, China
Yong-Lu Li received the PhD degree from Shanghai Jiao Tong University. He is a tenure-track Assistant Professor with Shanghai Jiao Tong University. Before joining SJTU, he was a postdoc fellow with HKUST and worked closely with Prof. CK Tang and Yu-Wing Tai. He was awarded the Baidu Scholarship, YunFan award, Shanghai Outstanding Graduate, etc. His research interests include computer vision, reasoning, and embodied AI.
Yong-Lu Li received the PhD degree from Shanghai Jiao Tong University. He is a tenure-track Assistant Professor with Shanghai Jiao Tong University. Before joining SJTU, he was a postdoc fellow with HKUST and worked closely with Prof. CK Tang and Yu-Wing Tai. He was awarded the Baidu Scholarship, YunFan award, Shanghai Outstanding Graduate, etc. His research interests include computer vision, reasoning, and embodied AI.View more
Author image of Xinpeng Liu
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Xinpeng Liu received the BE degree from Shanghai Jiao Tong University, Shanghai, China, where he is currently working toward the master's degree in computer science. His research interests include computer vision and deep learning.
Xinpeng Liu received the BE degree from Shanghai Jiao Tong University, Shanghai, China, where he is currently working toward the master's degree in computer science. His research interests include computer vision and deep learning.View more
Author image of Xiaoqian Wu
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Xiaoqian Wu is currently working toward the bachelor's degree with Shanghai Jiao Tong University, Shanghai, China. She is a research assistant with Machine Vision and Intelligence Group, Shanghai Jiao Tong University. Her research interests include computer vision and deep learning.
Xiaoqian Wu is currently working toward the bachelor's degree with Shanghai Jiao Tong University, Shanghai, China. She is a research assistant with Machine Vision and Intelligence Group, Shanghai Jiao Tong University. Her research interests include computer vision and deep learning.View more
Author image of Yizhuo Li
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Yizhuo Li received the BS degree in electronic engineering from Tsinghua University, Beijing, China, in 2019. He is currently working toward the master degree with Shanghai Jiao Tong University, supervised by Prof. Cewu Lu. His research interests mainly focus on computer vision.
Yizhuo Li received the BS degree in electronic engineering from Tsinghua University, Beijing, China, in 2019. He is currently working toward the master degree with Shanghai Jiao Tong University, supervised by Prof. Cewu Lu. His research interests mainly focus on computer vision.View more
Author image of Zuoyu Qiu
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Zuoyu Qiu is currently working toward the bachelor's degree with Shanghai Jiao Tong University, Shanghai, China. He is a research assistant with Machine Vision and Intelligence Group, Shanghai Jiao Tong University. His research interests include computer vision and deep learning.
Zuoyu Qiu is currently working toward the bachelor's degree with Shanghai Jiao Tong University, Shanghai, China. He is a research assistant with Machine Vision and Intelligence Group, Shanghai Jiao Tong University. His research interests include computer vision and deep learning.View more
Author image of Liang Xu
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Liang Xu received the master's degree in computer science from Shanghai Jiao Tong University, supervised by Prof. Cewu Lu. His research interests mainly focus on computer vision.
Liang Xu received the master's degree in computer science from Shanghai Jiao Tong University, supervised by Prof. Cewu Lu. His research interests mainly focus on computer vision.View more
Author image of Yue Xu
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Yue Xu received the BE degree from Shanghai Jiao Tong University, Shanghai, China, where he is currently working toward the master's degree in computer science. His research interests mainly include computer vision.
Yue Xu received the BE degree from Shanghai Jiao Tong University, Shanghai, China, where he is currently working toward the master's degree in computer science. His research interests mainly include computer vision.View more
Author image of Hao-Shu Fang
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Hao-Shu Fang received the BE degree in computer science from Shanghai Jiao Tong University, in 2019. He is currently working toward the PhD degree with the Department of Computer Science and Engineering, Shanghai Jiao Tong University. His research interests include computer vision and robot intelligence. He was awarded the CCF-CV Rising Scholar Award in 2019, Baidu Fellowship in 2019, MSRA Fellowship in 2020, and ByteDance Fellowship in 2020.
Hao-Shu Fang received the BE degree in computer science from Shanghai Jiao Tong University, in 2019. He is currently working toward the PhD degree with the Department of Computer Science and Engineering, Shanghai Jiao Tong University. His research interests include computer vision and robot intelligence. He was awarded the CCF-CV Rising Scholar Award in 2019, Baidu Fellowship in 2019, MSRA Fellowship in 2020, and ByteDance Fellowship in 2020.View more
Author image of Cewu Lu
Shanghai Digital Medicine Innovation Center, Ruijin Hospital, Shanghai, China
MoE Key Lab of AI, AI Institute, Shanghai Jiao Tong University, Shanghai, China
Shanghai Qi Zhi Institute, Shanghai, China
Cewu Lu received the PhD degree from the Chinese University of Hong Kong, supervised by Prof. Jiaya Jia. He is a professor with Shanghai Jiao Tong University. Before joining SJTU, he was a research fellow with Stanford University, working under Prof. Fei-Fei Li and Prof. Leonidas J. Guibas. He was a research assistant professor with the Hong Kong University of Science and Technology with Prof. Chi Keung Tang. His research interests fall mainly in computer vision and robotics.
Cewu Lu received the PhD degree from the Chinese University of Hong Kong, supervised by Prof. Jiaya Jia. He is a professor with Shanghai Jiao Tong University. Before joining SJTU, he was a research fellow with Stanford University, working under Prof. Fei-Fei Li and Prof. Leonidas J. Guibas. He was a research assistant professor with the Hong Kong University of Science and Technology with Prof. Chi Keung Tang. His research interests fall mainly in computer vision and robotics.View more

Contact IEEE to Subscribe

References

References is not available for this document.