Unified Transformation Framework of Open Tag Feature for Chinese Multi-modal Data

With the popularization of the Internet and the development of We Media, multi-modal data such as video, audio, image and text have become the mainstream forms of information dissemination. How to efficiently solve the representation heterogeneity problem of these multi-modal data has become a research hotspot. After studying a large amount of literature, it is found that the unified mapping of multi-modal data can be realized via text tags. In order to solve the problem of heterogeneous representation of multi-modal data, an open tag unified transformation framework for Chinese multi-modal data is proposed. This framework can use different conversion models for data of different modalities to convert them into text uniformly. High-quality multi-grained open text tags are then extracted from the above-mentioned converted texts using tag extraction algorithms. Finally, the data object corresponding to the source modality is mapped with the extracted text tags, so as to solve the transformation problem of unified tag feature of multi-modal data. The experimental results of each module can be fully explained that the technical feasibility of our framework, and provide a novel direction for feature transformation and representation of multi-modal data.