Joint Learning of Local and Global Context for Temporal Action Proposal Generation

Temporal action proposal generation is an important yet challenging problem, since temporal proposals with rich action content are indispensable for analysing real-world videos with long duration and high proportion irrelevant content. This problem requires methods not only generating proposals with precise temporal boundaries, but also retrieving proposals to cover ground truth action instances with high recall and high overlap using relatively fewer proposals. To address these difficulties, we introduce an effective and efficient proposal generation method, named Local-Global Network (LGN), by which local and global contexts are jointly learned to generate high quality proposals. Locally, LGN first locates temporal boundaries with high starting and ending probabilities separately, then directly combines these boundaries as proposals. Globally, LGN evaluates the actionness probability of multiple-durations temporal regions simultaneously using temporal convolutional layers and anchor mechanism. Finally, we combine the boundary probabilities of each proposal with actionness probability of matched temporal regions as the confidence score, which is used for retrieving proposals. We conduct experiments on two datasets: ActivityNet-1.3 and THUMOS-14, where LGN outperforms other state-of-the-art methods with both high recall and high temporal precision. Finally, further experiments demonstrate that by combining existing action classifiers, our method significantly improves the state-of-the-art temporal action detection performance.