Fusion of disparate information through joint embeddings

Most pattern recognition tasks can be abstracted to a problem of utilizing comparisons between objects to perform the given inference task. Often these comparisons are in the form of a distance measure or dissimilarity. The design of appropriate comparison functions for particular inference tasks is an area of extensive research, and often rests on expert knowledge of the problem domain. If the data of interest come from two different sensors, or consist of very different types of data, a single dissimilarity may be inappropriate; instead, one might utilize several dissimilarities, each designed for a specific sensor or data stream. In this work we consider the problem of fusing information obtained from very different sensors or sources, encoded through the use of dissimilarity functions. Given n observations from source j, we have an n × n dissimilarity measure D
_j
, and we wish to utilize all this information in our inference. We describe several methods of utilizing these dissimilarity matrices that are based on embedding the observations into a single space. These methods optimize either the fidelity (whether the distances in the embedded space match the original dissimilarities) or the commensurability (whether matched objects from different sensors are close in the embedded space) or both. We discuss the properties of these embeddings, apply the idea to a problem in network modeling, and point out some interesting areas of further research.