A Scalable Approach to Historical Data Management via Optimized Transaction Logs

Temporal databases play a crucial role in tracking the evolution of data over time by preserving historical states through valid time intervals instead of overwriting data. This approach enables robust data analysis, forecasting, and auditing while enforcing consistency and preventing conflicting entries. Traditional transactional systems complement this by recording the time context of changes in transaction logs, which are essential for ensuring data integrity, recovery, and auditability. However, conventional log management, relying on sequential scanning, imposes performance and scalability limitations. This paper introduces a novel, efficient solution for managing transaction logs through an integrated data mapping layer. The proposed structure stores direct references to logged data within the database, supported by five optimized indexes—including object definitions and four temporal B+trees—to track validity, load time, and transaction lifecycle. This block-oriented design enables dynamic access, supports prohibited history handling, and eliminates the need for separate temporal layers or data migrations. Performance evaluations demonstrate significant gains in processing speed and system scalability compared to traditional approaches. However, limitations persist, including the growing footprint of non-indexed log metadata, challenges in data recoverability, incompatibility with standard table-optimization techniques, static log block sizes, and potential bottlenecks during high-frequency change periods.