One Stone, Three Birds: Finer-Grained Encryption with Apache Parquet @ Large Scale

Data access control, retention, and encryption-at-rest are fundamental security goals for data privacy and compliance. Often each of these three goals are implemented independently and in various layers of the Big Data stack. For example, Big data query engines may add custom support for access control at the engine level but the underlying stored data is not necessarily secured. In the modern Data Lakehouse, there is open access to the underlying data in many companies’ data lakes and therefore it is not sufficient to implement security and compliance at an engine level.In this paper we present a unified way to address all three security goals at once through the shared lower layer of Apache Parquet which ensures these controls are enforced by higher level Big Data tools. We introduce performant file format level encryption with both column level and cell level granularity controlled by schema tagging. We achieve less than 8% write and less than 5% read performance overhead in most common scenarios of file format encryption by minimizing RPC calls required to perform encryption / decryption, leveraging AESNI and choosing optimal encryption options. Lastly we present a novel high throughput method for rewriting existing Apache Parquet data with encryption which is critical for practical onboarding of a data lake to Apache Parquet encryption.