On the characterization of the Oracle for dynamic classifier selection

The Oracle model has been used not only for comparison between techniques but also in the design of different methods in Multiple Classifier Systems (MCS). Even though the model represents the ideal classifier selection scheme, Dynamic Classifier Selection (DCS) techniques present a large performance gap from the Oracle. This means that, for a significant number of instances, the DCS techniques are not able to select a competent classifier, despite the Oracles assurance of its presence in the pool. Given that issue, this work aims to investigate the reasons why the Oracle model may not be well suited for guiding the search for a promising pool of classifiers for DCS techniques. For this purpose, a pool generation method that guarantees an Oracle accuracy rate of 100% in the training set is proposed. This method is further used to analyse the behavior of DCS techniques when the presence of at least one competent classifier in the pool for each training sample is assured. Experiments show that integrating Oracle information in the generation phase of an MCS has little impact on the gap between the accuracy rates of DCS techniques and the Oracle. Moreover, it is also shown that, for a theoretical limit of 100%, the DCS techniques were only able to select a competent classifier for at most 85% of the instances, on average. Results suggest that the Oracle is not the best guide for generating a pool of classifiers for DCS, for the model is performed globally whilst DCS techniques work with local data only.