LiteIndex: Memory-Efficient Schema-Agnostic Indexing for JSON documents in SQLite

SQLite with JSON (JavaScript Object Notation) format is widely adopted for local data storage in mobile applications such as Twitter and Instagram. With more data are generated and stored, it becomes vitally important to efficiently index and search JSON records in SQLite. However, current methods in SQLite either require full text search (that incurs big memory usage and long query latency) or indexing based on expression (that needs to be manually created by specifying search keys). On the other hand, existing JSON auto-matic indexing techniques, mainly focusing on big data and cloud environments, depend on a colossal tree structure that cannot be applied in memory-constrained mobile devices.In this paper, we propose a novel schema-agnostic indexing technique called LiteIndex that can automatically index JSON records by extracting keywords from long text and maintaining user-preferred items within a given memory constraint. This is achieved by memory-efficient index organization with light-weight keyword extraction from long text and user-preference-aware reinforcement-learning-based index pruning mechanism. LiteIndex has been implemented in a Android smartphone platform and evaluated with a dataset from Tweet. Experimental results show that LiteIndex can significantly reduce the query latency by up to 18x with less memory usage compared with SQLite with FTS3/FTS4 extensions.