A blackboard architecture for data-intensive information fusion using locality-sensitive hashing

The problem of identifying patterns in large data sets arises in applications such as the analysis of surveillance data as well as for more general situational awareness needs. Often input data arrives in bits and pieces, in random order from multiple sensors. Conjecturing high-level patterns from such data is often referred to as information fusion. Blackboard-based algorithms have been used to automatically identify such patterns. However, when the volume of data is large and includes noise, a naive blackboard-based algorithm can be inefficient due to the large number of combinations of inputs that a knowledge source could be applied on. Here we present an approach to improve the performance of blackboard-based algorithms under the assumption that each knowledge source is ‘locally-selective’. The number of combinations to explore can then be pruned using locality sensitive hashing (LSH) We formally define a generic Blackboard architecture using Bayesian knowledge sources, with which we model three problems including a real-life example of monitoring Twitter news feeds. We also present experimental results demonstrating the advantage of LSH over a naïve blackboard algorithm.