Multistage classification by cascaded classifiers

We propose a new method of classification built as a cascade of a distributed learner and a local learner. The distributed learner generalizes to learn the "rule" and the local learner learns the "exceptions" not covered by the "rule". We show how such a system can be trained using cross-validation. We use a multilayer perceptron with sigmoidal hidden units as the rule-learner and a k-nearest neighbor classifier as the exception-learner. Cascading is a better approach than voting where multiple learners are used for all cases; the extra computation and memory required for the second learner is unnecessary if we are sufficiently sure that the first one's response is correct. The cascade algorithm significantly outperforms the individual methods and voting on three optical and pen-based handwritten digit recognition tasks when comparison is based on three criteria; generalization success, learning speed, and number of free parameters.