Implementing conceptual search capability in a cloud-based feed aggregator

Modern society is producing more information at a faster pace than ever before. There is an increasing desire to use this massive amount of collected data to solve real world problems. But given the vast size of the data involved and the resource intensive nature of rapid large-data processing, the need for more advanced methodologies in this regard is increasing. This phenomenon has given rise to the term ‘Big Data’ which references these types of data intensive problems; problems that are typically beyond the ability of traditional tools and methodologies to effectively address. This paper documents the means by which a problem of this type has been addressed using agent-oriented software engineering methodologies and commercial cloud technology. The Feed Analyzer is a conceptual search based Web feed aggregation system deployed to Microsoft's Windows Azure cloud platform. Work of the type documented within this paper will become of greater important as cloud technology is increasingly used to address these types of data intensive problems.