Vision and Knowledge Bimodal Fine-grained Recognition Based on Hierarchical Structured Knowledge Extraction

In the field of fine-grained recognition, the most important method is to use visual information as the most important modality to complete the task. In order to prevent the model from being greatly affected by the data of a single modality, and to improve the accuracy and robustness and security, existing methods fused information such as text, voice or knowledge graphs on a single modality basis. In view of the limitations of the previous methods in the way of integrating knowledge modalities, this paper optimizes the existing knowledge extraction methods for bird datasets, and proposes a hierarchical and structured knowledge extraction method, so that the model can be classified when only knowledge modalities are used. The accuracy reaches 48.69%, which improves the accuracy by about 10% compared with previous methods. After multimodal fusion of knowledge extracted by this method and visual modalities, about 2% improvement was achieved on the knowledge-visual fusion framework KGRF, and the results proved the effectiveness of this method.