Cross Join Operation Over Large Quantities of Data Considering Data Locality and Data Replication

The research into data analysis over large quantities of data requires different strategies from traditional computation, in order to allow data processing in this scenario and extract valuable information from it. Cross join is a common operation from the relational algebra that other operations can be derived from it, like the Theta-join operation. Implementing it, however, implies excessive data traffic over network. We propose to use the cross join operation with input files, and use metadata information from the distributed filesystem such as data locality and data replication to distribute the operation among the machines that store parts of the files. The results showed that using this strategy, the data traffic necessary to the operation can be significantly reduced, increasing application performance.