On Classification Confidence and Ranking Using Decision Trees

In this paper a novel method is proposed that extends the decision tree framework, allowing standard decision tree classifiers to provide a unique certainty value for every input sample they classify. This value is calculated for every input sample individually and represents the classifier's certainty in the classification.The algorithm consists of three main parts. 1) The input sample's distance is calculated to the decision boundary. This step involves solving a set of linearly constrained quadratic programs. The distance calculating procedure also allows the use of different distance metrics, where the minimal distance projection is not necessarily invariant. 2) Kernel density estimation is done on the distance values of a training set to obtain conditional true and false classification profiles. 3) Using the conditional densities Bayesian computation is applied to calculate the conditional true classification probability, which we use as classification certainty. The algorithm proposed in this paper is not limited to axis parallel trees, it can be applied to any kind of decision tree where the decisions are hyperplanes (not necessarily parallel to the axes). The algorithm does not alter the tree structure, the growth process is not modified. It only uses the training data to obtain true and false classification profiles conditional to distance from the decision boundary. The usability of the method is demonstrated on two examples. One artificial two dimensional dataset, and one real world nine dimensional dataset. It is shown that the method can significantly increase the classification accuracy (in the cost of rejecting a certain number of samples, saying their classification would be too "risky"). It is also demonstrated that the classification certainty value can be effectively used for ranking purposes.