Nucleus Neural Network: A Data-driven Self-organized Architecture

In this paper, inspired from the nuclei in brain, we propose a nucleus neural network (NNN) and corresponding connecting architecture learning method. In a nucleus, the neurons are not assigned as regular layers, i.e., a neuron may connect to any neurons in the nucleus and the connections are self-organized according to data distribution. This type of architecture gets rid of layer limitation and makes full use of processing capability of each neuron. It is crucial to assign connections between all the neuron pairs. To address the time demanding of objectives in traditional architecture learning methods, we propose an efficient architecture learning model for the nucleus based on the principle that more relevant input and output neuron pair deserves higher connecting density. The new objective measures the information flow through the network architecture without involvement of weights and biases which greatly reduces the computational complexity. We find that this novel architecture is robust to irrelevant components in test data. So we reconstruct a new dataset based on the MNIST dataset where the types of digital backgrounds in training and test sets are different. The new dataset is a great challenge for learners because training data and test data are not only independent but also follow different distributions. Experiments demonstrate that NNN achieves significant improvement over architectures with regular layers on the reconstructed dataset.