Stacked generalization in neural networks: generalization on statistically neutral problems

Generalization continues to be one of the most important topic in neural networks and other classifiers. In the last number of years, number of different methods have been developed to improve generalization accuracy. Any classifier that uses induction to find the class concept from the training patterns will have a hard time to achieve an acceptable level of generalization accuracy when the problem to be learned is a statistically neutral problem. A problem is statistically neutral if the probability of mapping an input onto an output is always the chance value of 0.5. We examine the generalization behaviour of multilayer neural networks on learning statistically neutral problems using single level learning models (e.g., conventional cross-validation scheme) as well as multiple level learning models (e.g., stacked generalization method). We show that for statistically neutral problems such as parity and majority function, the stacked generalization scheme improves classification performance and generalization accuracy over the single level cross-validation model.