Topics and relationship discovery from unstructured equivocal sources for situation assessment

The proliferation of soft information sources has led to an explosion of data while creating fertile grounds to obtain actionable insights. In such an unstructured data environment, the fundamental challenge is to discover topics and latent relationships, indicating significant events, in an unsupervised manner from multiple sources. The extracted topics and relationships then provide evidence for enhanced model-based situation assessment (SA) if likely equivocal nature of multisource data is resolved appropriately. Our approach to topic extraction is an array of powerful algebraic and probabilistic text classification and topic extraction techniques to uncover relationships. This includes Latent Semantic Analysis (LSA) and Latent Dirichlet Allocation (LDA), and their generalizations. Our approach to relationship extraction is to apply deep linguistics processing techniques for information structuring and representation in the form of subject-predicate-object Resource Description Framework (RDF) triples. Detected significant events and relationships from the environment then may trigger invocations of various generative computational models for situation assessment, such as Bayesian Networks (BN). The fact that extracted topics and triples are in their simplest fundamental linguistic data structure, duplicate evidence from multiple sources will surface up in the same normalized form along with annotated credibility and reliability. Hence the process of resolving the equivocal issue will be an easier task. We will explain the proposed approach using powerful in-house text analytics software aText.