Abstract:
Human activity understanding is of widespread interest in artificial intelligence and spans diverse applications like health care and behavior analysis. Although there ha...Show MoreMetadata
Abstract:
Human activity understanding is of widespread interest in artificial intelligence and spans diverse applications like health care and behavior analysis. Although there have been advances with deep learning, it remains challenging. The object recognition-like solutions usually try to map pixels to semantics directly, but activity patterns are much different from object patterns, thus hindering another success. In this article, we propose a novel paradigm to reformulate this task in two-stage: first mapping pixels to an intermediate space spanned by atomic activity primitives, then programming detected primitives with interpretable logic rules to infer semantics. To afford a representative primitive space, we build a knowledge base including 26+ M primitive labels and logic rules from human priors or automatic discovering. Our framework, Human Activity Knowledge Engine (HAKE), exhibits superior generalization ability and performance upon canonical methods on challenging benchmarks. Code and data are available at http://hake-mvig.cn/.
Published in: IEEE Transactions on Pattern Analysis and Machine Intelligence ( Volume: 45, Issue: 7, 01 July 2023)
Funding Agency:
Qing Yuan Research Institute, Shanghai Jiao Tong University, Shanghai, China
Yong-Lu Li received the PhD degree from Shanghai Jiao Tong University. He is a tenure-track Assistant Professor with Shanghai Jiao Tong University. Before joining SJTU, he was a postdoc fellow with HKUST and worked closely with Prof. CK Tang and Yu-Wing Tai. He was awarded the Baidu Scholarship, YunFan award, Shanghai Outstanding Graduate, etc. His research interests include computer vision, reasoning, and embodied AI.
Yong-Lu Li received the PhD degree from Shanghai Jiao Tong University. He is a tenure-track Assistant Professor with Shanghai Jiao Tong University. Before joining SJTU, he was a postdoc fellow with HKUST and worked closely with Prof. CK Tang and Yu-Wing Tai. He was awarded the Baidu Scholarship, YunFan award, Shanghai Outstanding Graduate, etc. His research interests include computer vision, reasoning, and embodied AI.View more
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Xinpeng Liu received the BE degree from Shanghai Jiao Tong University, Shanghai, China, where he is currently working toward the master's degree in computer science. His research interests include computer vision and deep learning.
Xinpeng Liu received the BE degree from Shanghai Jiao Tong University, Shanghai, China, where he is currently working toward the master's degree in computer science. His research interests include computer vision and deep learning.View more
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Xiaoqian Wu is currently working toward the bachelor's degree with Shanghai Jiao Tong University, Shanghai, China. She is a research assistant with Machine Vision and Intelligence Group, Shanghai Jiao Tong University. Her research interests include computer vision and deep learning.
Xiaoqian Wu is currently working toward the bachelor's degree with Shanghai Jiao Tong University, Shanghai, China. She is a research assistant with Machine Vision and Intelligence Group, Shanghai Jiao Tong University. Her research interests include computer vision and deep learning.View more
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Yizhuo Li received the BS degree in electronic engineering from Tsinghua University, Beijing, China, in 2019. He is currently working toward the master degree with Shanghai Jiao Tong University, supervised by Prof. Cewu Lu. His research interests mainly focus on computer vision.
Yizhuo Li received the BS degree in electronic engineering from Tsinghua University, Beijing, China, in 2019. He is currently working toward the master degree with Shanghai Jiao Tong University, supervised by Prof. Cewu Lu. His research interests mainly focus on computer vision.View more
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Zuoyu Qiu is currently working toward the bachelor's degree with Shanghai Jiao Tong University, Shanghai, China. He is a research assistant with Machine Vision and Intelligence Group, Shanghai Jiao Tong University. His research interests include computer vision and deep learning.
Zuoyu Qiu is currently working toward the bachelor's degree with Shanghai Jiao Tong University, Shanghai, China. He is a research assistant with Machine Vision and Intelligence Group, Shanghai Jiao Tong University. His research interests include computer vision and deep learning.View more
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Liang Xu received the master's degree in computer science from Shanghai Jiao Tong University, supervised by Prof. Cewu Lu. His research interests mainly focus on computer vision.
Liang Xu received the master's degree in computer science from Shanghai Jiao Tong University, supervised by Prof. Cewu Lu. His research interests mainly focus on computer vision.View more
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Yue Xu received the BE degree from Shanghai Jiao Tong University, Shanghai, China, where he is currently working toward the master's degree in computer science. His research interests mainly include computer vision.
Yue Xu received the BE degree from Shanghai Jiao Tong University, Shanghai, China, where he is currently working toward the master's degree in computer science. His research interests mainly include computer vision.View more
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Hao-Shu Fang received the BE degree in computer science from Shanghai Jiao Tong University, in 2019. He is currently working toward the PhD degree with the Department of Computer Science and Engineering, Shanghai Jiao Tong University. His research interests include computer vision and robot intelligence. He was awarded the CCF-CV Rising Scholar Award in 2019, Baidu Fellowship in 2019, MSRA Fellowship in 2020, and ByteDanc...Show More
Hao-Shu Fang received the BE degree in computer science from Shanghai Jiao Tong University, in 2019. He is currently working toward the PhD degree with the Department of Computer Science and Engineering, Shanghai Jiao Tong University. His research interests include computer vision and robot intelligence. He was awarded the CCF-CV Rising Scholar Award in 2019, Baidu Fellowship in 2019, MSRA Fellowship in 2020, and ByteDanc...View more
Shanghai Digital Medicine Innovation Center, Ruijin Hospital, Shanghai, China
MoE Key Lab of AI, AI Institute, Shanghai Jiao Tong University, Shanghai, China
Shanghai Qi Zhi Institute, Shanghai, China
Cewu Lu received the PhD degree from the Chinese University of Hong Kong, supervised by Prof. Jiaya Jia. He is a professor with Shanghai Jiao Tong University. Before joining SJTU, he was a research fellow with Stanford University, working under Prof. Fei-Fei Li and Prof. Leonidas J. Guibas. He was a research assistant professor with the Hong Kong University of Science and Technology with Prof. Chi Keung Tang. His research...Show More
Cewu Lu received the PhD degree from the Chinese University of Hong Kong, supervised by Prof. Jiaya Jia. He is a professor with Shanghai Jiao Tong University. Before joining SJTU, he was a research fellow with Stanford University, working under Prof. Fei-Fei Li and Prof. Leonidas J. Guibas. He was a research assistant professor with the Hong Kong University of Science and Technology with Prof. Chi Keung Tang. His research...View more
Contents Qing Yuan Research Institute, Shanghai Jiao Tong University, Shanghai, China
Yong-Lu Li received the PhD degree from Shanghai Jiao Tong University. He is a tenure-track Assistant Professor with Shanghai Jiao Tong University. Before joining SJTU, he was a postdoc fellow with HKUST and worked closely with Prof. CK Tang and Yu-Wing Tai. He was awarded the Baidu Scholarship, YunFan award, Shanghai Outstanding Graduate, etc. His research interests include computer vision, reasoning, and embodied AI.
Yong-Lu Li received the PhD degree from Shanghai Jiao Tong University. He is a tenure-track Assistant Professor with Shanghai Jiao Tong University. Before joining SJTU, he was a postdoc fellow with HKUST and worked closely with Prof. CK Tang and Yu-Wing Tai. He was awarded the Baidu Scholarship, YunFan award, Shanghai Outstanding Graduate, etc. His research interests include computer vision, reasoning, and embodied AI.View more
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Xinpeng Liu received the BE degree from Shanghai Jiao Tong University, Shanghai, China, where he is currently working toward the master's degree in computer science. His research interests include computer vision and deep learning.
Xinpeng Liu received the BE degree from Shanghai Jiao Tong University, Shanghai, China, where he is currently working toward the master's degree in computer science. His research interests include computer vision and deep learning.View more
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Xiaoqian Wu is currently working toward the bachelor's degree with Shanghai Jiao Tong University, Shanghai, China. She is a research assistant with Machine Vision and Intelligence Group, Shanghai Jiao Tong University. Her research interests include computer vision and deep learning.
Xiaoqian Wu is currently working toward the bachelor's degree with Shanghai Jiao Tong University, Shanghai, China. She is a research assistant with Machine Vision and Intelligence Group, Shanghai Jiao Tong University. Her research interests include computer vision and deep learning.View more
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Yizhuo Li received the BS degree in electronic engineering from Tsinghua University, Beijing, China, in 2019. He is currently working toward the master degree with Shanghai Jiao Tong University, supervised by Prof. Cewu Lu. His research interests mainly focus on computer vision.
Yizhuo Li received the BS degree in electronic engineering from Tsinghua University, Beijing, China, in 2019. He is currently working toward the master degree with Shanghai Jiao Tong University, supervised by Prof. Cewu Lu. His research interests mainly focus on computer vision.View more
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Zuoyu Qiu is currently working toward the bachelor's degree with Shanghai Jiao Tong University, Shanghai, China. He is a research assistant with Machine Vision and Intelligence Group, Shanghai Jiao Tong University. His research interests include computer vision and deep learning.
Zuoyu Qiu is currently working toward the bachelor's degree with Shanghai Jiao Tong University, Shanghai, China. He is a research assistant with Machine Vision and Intelligence Group, Shanghai Jiao Tong University. His research interests include computer vision and deep learning.View more
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Liang Xu received the master's degree in computer science from Shanghai Jiao Tong University, supervised by Prof. Cewu Lu. His research interests mainly focus on computer vision.
Liang Xu received the master's degree in computer science from Shanghai Jiao Tong University, supervised by Prof. Cewu Lu. His research interests mainly focus on computer vision.View more
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Yue Xu received the BE degree from Shanghai Jiao Tong University, Shanghai, China, where he is currently working toward the master's degree in computer science. His research interests mainly include computer vision.
Yue Xu received the BE degree from Shanghai Jiao Tong University, Shanghai, China, where he is currently working toward the master's degree in computer science. His research interests mainly include computer vision.View more
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Hao-Shu Fang received the BE degree in computer science from Shanghai Jiao Tong University, in 2019. He is currently working toward the PhD degree with the Department of Computer Science and Engineering, Shanghai Jiao Tong University. His research interests include computer vision and robot intelligence. He was awarded the CCF-CV Rising Scholar Award in 2019, Baidu Fellowship in 2019, MSRA Fellowship in 2020, and ByteDance Fellowship in 2020.
Hao-Shu Fang received the BE degree in computer science from Shanghai Jiao Tong University, in 2019. He is currently working toward the PhD degree with the Department of Computer Science and Engineering, Shanghai Jiao Tong University. His research interests include computer vision and robot intelligence. He was awarded the CCF-CV Rising Scholar Award in 2019, Baidu Fellowship in 2019, MSRA Fellowship in 2020, and ByteDance Fellowship in 2020.View more
Shanghai Digital Medicine Innovation Center, Ruijin Hospital, Shanghai, China
MoE Key Lab of AI, AI Institute, Shanghai Jiao Tong University, Shanghai, China
Shanghai Qi Zhi Institute, Shanghai, China
Cewu Lu received the PhD degree from the Chinese University of Hong Kong, supervised by Prof. Jiaya Jia. He is a professor with Shanghai Jiao Tong University. Before joining SJTU, he was a research fellow with Stanford University, working under Prof. Fei-Fei Li and Prof. Leonidas J. Guibas. He was a research assistant professor with the Hong Kong University of Science and Technology with Prof. Chi Keung Tang. His research interests fall mainly in computer vision and robotics.
Cewu Lu received the PhD degree from the Chinese University of Hong Kong, supervised by Prof. Jiaya Jia. He is a professor with Shanghai Jiao Tong University. Before joining SJTU, he was a research fellow with Stanford University, working under Prof. Fei-Fei Li and Prof. Leonidas J. Guibas. He was a research assistant professor with the Hong Kong University of Science and Technology with Prof. Chi Keung Tang. His research interests fall mainly in computer vision and robotics.View more
