Semantic Information in Gating Patterns of Dynamic Convolutional Neural Networks

Dynamic Convolutional Neural Networks are an emerging class of models characterized by their ability to dynamically adjust inference complexity at run-time, by identifying parts of the model with minimal contribution to the result and skipping the corresponding computations. A prominent such category includes models that generate binary gating signals indicating whether specific convolutional kernels need to be computed or can be omitted based on the characteristics of each processed datum. These signals are usually generated by branches of the same model which are typically learned simultaneously to the main task, with their main objective being to enable good performance with parsimony of computations. We argue that such objective incentivizes the model to implicitly optimize and utilize kernels in class/concept –specific groups, hence ascribing semantic information to the gating signals. We demonstrate this behavior by studying the characteristics of such signals for popular CNN architectures in the ImageNet database. By comparing the relationship between gating signals from different visual categories in the ImageNet hierarchy, it is shown that the gating patterns’ dissimilarity correlates well with semantic span of the underlying classes. It is also demonstrated that through appropriate distance measures, gating patterns can be used for ranking classes’ similarity with comparable performance to that off standard CNN-generated image descriptors, but in a significantly more compact representation due to their binary nature. (Abstract)