Cost-effective Modality Selection for Video Popularity Prediction

Video popularity prediction, also known as predicting the future popularity of a video, is a task that generally uses various modalities of the video, such as visual, audio, text, and metadata, to make predictions. However, the traditional approach puts all the modalities directly into a model for processing, which introduces noise and redundancy into the model. To address this issue, we propose Cost-effective Modality Selection(CeMS) for video popularity prediction, which can adaptively select cost-effective modalities as input based on the features of the metadata. Specifically, a policy network first obtains the prior information from metadata to analyze the differences in representation ability and computation cost between various modalities. Then, with the policy estimation, an optimal set of cost-effective modalities is determined, and the backbone network corresponding to the selected modalities is activated for feature extraction. Finally, the semantic information of all selected modalities is fed into the decoder to yield the predicted popularity. Experiments demonstrate that our proposed approach yields 63%-69% reduction in computation when compared to the traditional baseline that simply uses all the modalities. Additionally, we achieve consistent improvements in accuracy over the state-of-the-art methods.