Inhibitory synapses in neural networks with sigmoidal nonlinearities

We study the behavior of anti-Hebbian synapses at a neural node that contains the standard sigmoidal nonlinearity. The criterion generalizes the idea already discussed by one of the authors for linear networks and consists in removing the correlation between the input to the synapse, and the node output. We show how the solution, just as for the linear case, is unique and can be learned with a standard anti-Hebbian rule. We suggest how these synapses can be embedded in fully self-organizing networks to generate orthogonal nonlinear components and be used for multidimensional approximation.