Faster Querying for Database Integration and Virtualization with Distributed Semi-Joins

Data integration and virtualization is commonly used to combine data for data analytics and reporting. A major challenge is handling large data sizes (“Big Data”) as moving data across a network is extremely expensive and limits query processing. Business intelligence and data visualization software require rapid response times for users, and data virtualization is often limited for use cases involving joins across systems. The contribution of this work is a semi-join based approach to data virtualization joins that minimizes data movement and utilizes the extensive resources available in the database systems rather than performing query processing in the virtualization engine. The result is significantly less data movement which translates into faster query times and higher performance. Experimental results demonstrate that performance can be increased by an order of magnitude. A unique feature of the approach is that it does not require any special software installed above the database servers such as mediators and works directly using SQL queries.