A High-Performance Scientific Database Supporting In-situ Data Query and Accessing

Scientific databases have become an increasingly popular and important asset in cloud computing, HPC, big data, and AI, which many applications rely on. The performance of querying on scientific databases is crucial as applications frequently select a specific subset of the database to work on based on certain attribute values. Current solutions support scientific databases by combining relational databases (RDB) and distributed file systems (DFS). However, in production environments, they usually cannot be collocated on the same cluster. To make matters worse, the production DFS’s dynamic namespace prevents the RDB from storing DFS file paths as the value. This results in remote file copying from the RDB server to the production DFS where computation programs load data, greatly limiting the system’s overall performance. To this end, this paper presents SEAFS, a file system that supports efficient searching on file extended attributes. First, SEAFS proposes an integrated indexed attribute storage scheme using a three-level search tree and weak link semantics to organize attributes directly within the file system namespace, enabling redundancy-free storage and rapid attribute searching. Second, SEAFS supports fuzzy searching with a per-attribute dedicated index using general suffix automata to provide optimized matching performance. Evaluations demonstrate that SEAFS outperforms the current solution (without remote copying) by 25.98–79.40× and 2.58–2271× in multi-threaded precise searching and fuzzy searching respectively.