On the information capacity of hinge functions

So called hinge functions play an important role in many applications, e.g., deep learning, support vector machines, regression, classification and others. A thorough theory, which explains why some of these applications are so successful for their respective purpose, is still missing. This paper aims at filling a knowledge gap by answering the question of how much information can be conveyed across a neural node, e.g., which uses the hinge loss function on weighted agglomerated information from precedent nodes. We hope that insight from artificial neural networks may also have implications for understanding biological information processing. As key results in this paper, an elegant representation of mutual information is derived and, furthermore, some structural properties of a channel, that consists of a certain input signal which is overlaid by additive noise and is filtered by a hinge function, are investigated. Determining the capacity of this channel in an explicit form, although an important fundamental problem, seems to be extremely difficult and unsolved as of today. Thus, necessary and sufficient conditions for an optimal input signal along with upper bounds on the capacity are deduced. Furthermore, we conjecture that exponentially distributed input signals are asymptotically capacity-achieving in the high SNR regime.