Distributed big data search for analyst queries and data fusion

We describe here an agent-based Distributed Analytical Search (DAS) tool to search and query distributed “big data” sources regardless of data's location, content or format. DAS semantically analyzes natural language queries from a web-based user interface. It automatically translates the query to a set of sub-queries by deploying a combination of planning and traditional database query optimization techniques. It then generates a query plan represented in XML and guide the execution by spawning intelligent agents with various types of wrappers as needed for distributed sites. The answers returned by the agents are merged appropriately and return them to the user. We have demonstrated DAS using a variety of data sources that are distributed and heterogeneous. DAS is the prime target for analysts searching relevant data sources to answer priority intelligence requirements without having them to know the details of available data sources. DAS enables fusion systems to search relevant data sources and extract evidence to propagate into the models of the systems.