Maintaining temporal coherency of virtual data warehouses

In electronic commerce applications such as stock trading, there is a need to consult sources available on the web for informed decision making. Because information such as stock prices keep changing, the web sources must be queried continually to maintain temporal coherency of the collected data, thereby avoiding decisions based on stale information. However because network infrastructure has failed to keep pace with ever growing web traffic, the frequency of contacting web servers must be kept to a minimum. This paper presents adaptive approaches for the maintenance of temporal coherency of data gathered from web sources. Specifically, it introduces mechanisms to obtain timely updates from web sources, based on the dynamics of the data and the users' need for temporal accuracy by judiciously, combining push and pull technologies and by using virtual data warehouses to disseminate data within acceptable tolerance to clients. A virtual warehouse maintains temporal coherence, within the tolerance specified, by tracking the amount of change in the web sources, pulling the data from the sources at opportune times, and pushing them to the clients according to their temporal coherence requirements. The performance of these mechanisms is studied using real stock price traces. One of the attractive features of these mechanisms is that it does not require changes to either the web servers or to the HTTP protocol.