Adapting CakeDB to Integrate High-Pressure Big Data Streams with Low-Pressure Systems

Big Data continues to be one of the hottest topics in the computer science field and itself takes many forms. One way Big Data manifests is in the form of streams. These streams can be generally defined by their update frequency and the bandwidth they consume. They can however be further defined by the characteristics of the data they carry. The producers of these streams are generally tuned to perform a given role (such as moving large quantities of data with low latency) which can often be at odds with the requirements of a given consumer. In many cases the logistics of consuming such a stream can make the task impractical. This paper discusses the concept of data streams as sequential data sets and having different pressures. The paper demonstrates through a use case of a financial trading company and a High Performance Compute Cluster how different applications require different pressures and why it is necessary to be able to scale down high pressure streams for low pressure applications without impacting the applications that require the full high pressure feed and the high pressure feed itself. A proposed system for classifying streams and related consumers is discussed as well as the concept of conflation as it applies to these data streams. Features in the prototype stream oriented database (CakeDB) that support adapting high-pressure streams to low-pressure applications are then discussed and further work is identified.