Building a Real-Time Web Observatory

Real-time data streams are becoming primary ways to access data from the Web and from Internet-ready devices. Real-time streams, personal devices, and sensor networks have the potential to contain rich insights for researchers, commerce, and governments. With a vested interest in unlocking the potential benefits hidden within, there has been extensive work conducted on developing technologies to process, integrate, and extract value from the data. However, exposing the value in this data is achieved via sharing the data in a secure and controllable environment. To that end, here the authors present the Web Observatory, a Web platform with an architecture capable of harvesting, querying, and analyzing multiple real-time and historic heterogeneous data, while providing data owners access control to their resources. They consider the current landscape and challenges of using data, analytics, and visualizations, and describe a series of use cases for the Web Observatory.