Agile Composition of Compliant Data Analytics Platforms

Sensitive data such as health records and financial transactions are increasingly being stored and processed in the cloud. Correspondingly, laws and regulations have been established to protect such data. For a cloud-based analytics service provider, it is of paramount importance to protect the sensitive information contained in customer data, while running analytics on it. While there exist a plethora of technologies to safeguard data, regulatory rules are not always defined in clear technical terms, and different regulations may impose different (or sometimes conflicting) rules on the analytics platform. Therefore, it remains a challenge in developing a platform that can support various security and compliance-enabling mechanisms, in a agile fashion, to reduce maintenance effort as well as improving scalability and performance. To address this challenge, we introduce the design and implementation of a cloud-based middleware platform that supports on-demand composition and configuration of security mechanisms to ease regulatory compliance enablement. We discuss at length our experiences and lessons learned from using our platform to deploy secure analytics systems at IBM and highlight the benefits of our approach by discussing the performance impact and trade-offs of different security mechanisms with respect to regulatory compliance.