Implicit order join: Joining log data with property data by discovering implicit order-oriented keys with human assistance

Data integration is still laboursome task when integrating data are not consistently managed. Such inconsistency can happen easily in real-world situations, such as properties of objects are managed by a central organization and trajectories (or logs) of the objects are recorded by other peripheral organizations. This paper deals with a case of missing ordering information. Integrating property data and log data without ordering information causes duplicated results. In order to solve this problem, this paper proposes a join algorithm, called implicit order join, which discovers implicit ordering information from both property data and log data with help of partial true integrated results from human assistance. With the discovered ordering information, the implicit order join enables to integrate the property data and log data. In order to discover the implicit ordering information, ordering correlation between attribute sequences of property data and log data should be found from comprehensive examination of possible attribute sequence pairs. The potential number of sequence pairs is as high as factorial order of the number of attributes. Therefore, this paper develops a heuristic approach to prune unnecessary examinations based on ordering dependency between attribute sequences. Experimental evaluation in this paper indicates that implicit order join can reduce 77% labouring tasks for integration and the pruning method reduces the number of attribute sequences in orders of magnitude.