Synergetic Learning System Based on Swarm of Non-Gradient Learners

Non-gradient descent learning schemes have recently gained attention for their efficient training and simplification of learning control parameter tuning. However, most methods are designed for tabular data (or image data represented in the form of 1D feature vectors), which limits their ability to capture the spatial and semantic richness of structured data. Additionally, to enhance representation learning capacity, these models often struggle to accurately determine the required complexity before construction, leading to unnecessary model complexity. To address these issues, this paper proposes a Synergic Learning System based on Swarm of Non-Gradient Learners (SONG). The system introduces a neural network-based model composed of lightweight, efficient, and scalable learning units, including Pseudoinverse Learners (PILers), a learnable Patch Respective Field (PRF), and a Pseudoinverse Learning-based Autoencoder (PILAE) based MLP, enhancing representation learning for both tabular and image data. Inspired by synergetic learning system and emergence theory, SONG dynamically aggregates multiple base models, adaptively determining subsystem quantity and hierarchical depth. Experimental results show that SONG outperforms existing competitive baseline methods on several publicly available benchmark datasets, achieving efficient training, low hyperparameter sensitivity, and superior generalization, while generally exhibiting emergent phenomena.