Activation Control of Multiple Piecewise Linear Neural Networks

Piecewise linear neural networks (PLNNs) are proven universal approximators for continuous functions on the compact domain. For multiple PLNNs (mPLNNs) differing from each other in suffering different approximation errors (AEs) when approximating the same continuous function, activating all the PLNNs to approximate the continuous function, and then, picking up the minimum AE (MAE) from all the AEs seems to be a practical way to arrive at such MAE. Activating PLNNs has to consume energy, and more activated PLNNs provide more AEs to consider, which can maximize the probability of guaranteeing the MAE, while also needing to harvest more energy. Therefore, how to make the optimal tradeoff between energy harvested and approximation accuracy for mPLNNs arises as an interesting issue. To address this problem, this paper first deduces the objective function, with the accumulative probability of obtaining the MAE as the objective and the accumulative energy harvested as the constraint, then reveals the optimal probability that each hidden neuron (HN) in the activated PLNN should be activated, and finally uncovers the optimal activation probability for each PLNN. An algorithm based on our discovered theoretical results is proposed for mPLNNs to enjoy the maximum probability of achieving the MAE at the acceptable level of accumulative energy harvested. Theoretical analysis and experiments verify its performance.
Note to Practitioners
—This paper addresses the interesting issue of how to make mPLNNs approximate the continuous function with the highest approximation accuracy under the energy constraint. Through the insightful discovery of the optimal activation probability for each HN in the activated PLNN as well as the optimal activation probability for each PLNN, this paper facilitates mPLNNs to suffer the MAE in their approximations to continuous functions, with the maximum probability while maintaining the energy harvested within a given range. Because PLNNs can model the nonlinear and complex system with the arbitrary AE, our proposed approach can be effectively applied by such system to operate in an energy-efficient and universal-approximation manner, which we believe could push the development of the low-power multiple neural networks.