Fundamentals, Challenges, and Advances of Statistical Learning for Knowledge Discovery and Problem Solving: A BYY Harmony Perspective

In this paper, an intelligent system is featured by both its abilities of interpreting what are observed via discovering knowledge about the world it survives, and Its problem solving skills of handling each issue encountered in the world. Correspondingly, the abilities and skills are obtained by two types of learning via evidences or data from the world. Due to noises in observation and a finite size of samples, learning is statistical in nature, which faces two key challenges. One is finding appropriate mathematical representations to suit various dependence structures underlying world. The other is getting a good theory to guide learning such that dependence structures are not only learned into mathematical representations but also with an appropriate complexity that matches the size of samples (ie., learning reliable structures of underlying world). This paper consists of part parts. The first two parts summarize typical dependence structures for tackling the challenge one and typical learning theories for tackling for tackling the challenge two. The third part introduces Bayesian Ying Yang (BYY) system as a general framework that unifies typical dependence structures and BYY harmony learning for the challenge two, with several favorable features. To illustrate this BYY learning, in the fourth part we further introduce fundamentals of independence subspaces and advances obtained from BYY harmony learning on typical independence subspaces, including PCA, MCA, DCA, ICA, FA, TFA, NFA, BFA, LMSER, as well as their temporal extensions.