Neighborhood entropy

Measures of relevance between features play an important role in classification and regression analysis. Mutual information has been proved to be an effective measure for categorical features. However, there is a limitation in computing relevance between numerical features with mutual information. In this work, we generalize Shannon's information entropy to neighborhood information entropy and propose a measure of neighborhood mutual information. It is shown that the new measure is a natural extension of classical mutual information which reduces to the classical one if features are discrete; thus the new measure can also be used to compute the relevance between discrete variables. In experiment, we show that neighborhood mutual information produces the nearly same outputs as mutual information. However, unlike mutual information, no discretization is required in computing relevance when used the proposed algorithm.