OCF-HP: An Offline Compaction Framework Based on Hotspot Prediction for Data Lakehouse

With the rapid advancement of information technologies, the global data volume is increasing at an unprecedented speed. Data Lakehouse has become an effective solution for real-time aggregation of multimodal heterogeneous data and supporting multiple query modes. It is worth noting that due to the column-based storage commonly employed in Data Lakehouse and the resulting mechanisms such as Copy-on-Write and Merge-On-Read, data query speed, data freshness, and storage costs have become huge challenges. In this paper, an Offline Compaction Framework based on Hotspot Prediction (OCF-HP) was designed to address the above challenges. Specifically, we analyzed the Timeline data in Hudi which is the mainstream Data Lakehouse framework, then predicted the hotspot periods of data writing based on the improved LSTM model using such data, and realized automatic trigger of Hudi offline compaction tasks in reasonable non-hot periods. The experiment results show that OCF-HP provides Hudi with a more optimized offline compaction solution so that data lakehouse managers could achieve better query speed and data freshness with reducing storage costs in Hudi's read optimized mode.