Stream Data Management Based on Integration of a Stream Processing Engine and Databases

Developments in network and sensor device technologies enable us to easily obtain real-world information, such as locations of moving objects, weather information, news, and stock prices. These data are continuously supplied, and they are regarded as data streams. Because of the dramatical increase of streaming data, their management and utilization has become more and more important. This paper describes a data stream management system named Harmonica. Harmonica employs an architecture combining our stream processing engine named stream-spinner and relational DBMSs. Based on the architecture, the system processes both continuous queries and traditional one-shot queries. Moreover, Harmonica supports continuous persistence requirements for streaming data as well as queries including selection, join, projection, and user-defined functions over data streams. Users can also specify continuous queries that integrate streaming data and persistent data stored in databases. Using the Harmonica API, users can develop a variety of applications coping with different continuous steaming data and data stored in databases. Our system can be deployed in network environments to achieve efficient and dependable distributed stream processing.