Optimal Activation Functions via Causal Wyner-Ziv Coding

We propose a new method for deriving optimal activation functions that relate the strength of a neuron's stimulus to its firing intensity. Our method is based on an information-theoretic model of neuronal communication and yields activation functions that provably maximize the communication efficiency at a prespecified fidelity. Crucially, we show that such activation functions exist (under a mild assumption on the required fidelity) and provide an example thereof.