Multi-View Representation Learning Regularizer with Gaussian-Product Mixture Prior

We study the problem of distributed multi-view representation learning. In this problem, two agents observe each one distinct, but possibly statistically correlated, view and independently extract from it a suitable representation so that a decoder that gets both representations correctly estimates the hidden label. In the absence of any explicit coordination between the agents, a central question is: what should each agent extract from its view that is necessary and sufficient for a correct estimation at the decoder? In this paper, we investigate this question from a generalization error perspective. First, we establish generalization bounds in terms of the relative entropy between the distribution of the representations extracted from training and “test” datasets and a data-dependent symmetric prior, i.e., the joint Minimum Description Length (MDL) of the extracted representations. Then, we use the obtained bound to devise a regularizer; and investigate in depth the question of the selection of a suitable prior. In particular, we show that the selection of the joint prior as Gaussian-product mixture, which induces Gaussian mixture marginal prior for each marginal view, implicitly encourages the agents to extract and output redundant features; a finding which is somewhat counter-intuitive. Interestingly, we show that a joint weighted attention mechanism emerges naturally in our approach. Finally, we conduct several experiments that validate the efficiency of our proposed method.