A review study of human-affection knowledge on usability engineering

An increasingly large amount of multimodal content is posted on social media websites such as YouTube and Facebook every day. In order to cope with the growth of such so much multimodal data, there is very urgent need to develop an intelligent multi-modal analysis framework that can effectively extract information from multiple modalities. In this research work, here propose a novel multimodal information extraction agent, which infers and aggregates the semantic and affective information associated with user generated multimodal data in contexts such as e-learning, e-health, automatic video content tagging and human-computer interaction (HCI). In particular, the developed intelligent agent adopts an ensemble feature extraction approach by exploiting the joint use of tri-modal (text, audio and video data) features to enhance the multimodal information extraction process. In preliminary experiments using the INTERFACE and SEMINE dataset, our proposed multi-modal system is shown to achieve an accuracy of 88.75%, outperforming the best state-of-the-art system by more than 10%, or in relative terms, a 46% reduction in error rate.