Gaussian Distribution-Aware Commonsense Knowledge Learning for Scene Graph Generation

Knowledge-based Scene Graph Generation (SGG) requires external commonsense knowledge beyond the visual scene to infer the relation between objects. Such knowledge can be obtained in a variety of forms, such as vision, text, and graph. However, there are two drawbacks as follows: 1) Commonsense knowledge essentially has uncertainty, but current works usually represent knowledge in a deterministic manner, which is not well matched to its nature. 2) Using commonsense knowledge without denoising will introduce irrelevant information. This can increase the burden on the relation classifier and only obtain marginal gains over a large amount of data. In this paper, we propose a novel Gaussian distribution-aware commonsense knowledge learning method for SGG. First, we associate each object pair with a Gaussian distribution, which parametrizes visual context and commonsense as mean and variance, respectively. We prove that Gaussian modeling can provide a probabilistic soft space to measure the uncertainty of external knowledge, which allows diverse predictions. Second, to reduce semantic noise in commonsense, we sample multiple variables from the Gaussian distribution and train multi-expert classifiers, which can be dynamically examined for the ensemble softmax classification. Extensive comparative experiments on two benchmarks confirm that our method can achieve competitive performance against the state-of-the-art. Ablation studies verify the essential roles of individual components. Moreover, the visualization of multi-expert classifiers confirms our ability to integrate commonsense for relation inference.