PolyFuse: A Large-Scale Hybrid Data Fusion System

The big data era brought us petabytes of data together with the challenges of storing and efficiently accessing largescale datasets. However, it unexpectedly surprised everyone with an enormous variety of data sources and types, and corresponding different data models. Dealing with a variety of those data models turned out to be a “hard nut to crack” for almost all existing data management engines. Data integration, a mature field addressing problems of accessing and fusing data residing in more than one datasource, over the years came up with feasible semi-automatic solutions, most of which efficiently handle a handful of data sources represented in one or two different data models (e.g. relational and semistructured) [Haas et al., 2005], [Gubanov et al., 2008]. While this is significant progress, most of the solutions do not easily scale up, since they usually require some sort of human assistance, infeasible at scale. Unified Famous Object (UFO) [Gubanov et al., 2009], [Bellahsene et al., 2011], [Gubanov et al., 2011a] was one of the first attempts to crack data fusion at scale by introducing a new abstraction called UFO that could incrementally learn different representations of data objects in different sources, and, over time, train itself to recognize and map them with high accuracy without supervision. Having trained many such UFOs, the system would get more and more powerful, as it learned to recognize and access data objects across many sources without supervision. While UFO was definitely progress towards scaling up data fusion, it was not a “silver bullet”, because it was built mostly for relational data. Hence, there is a need for a new large-scale data integration system that could handle many types of data at scale. This paper sketches its architecture, and envisions potential research challenges. First, a few key principles that such a system should follow are described, then follows the discussion on architecture and research avenues.