On Guo and Nixon's Criterion for Feature Subset Selection: Assumptions, Implications, and Alternative Options

Guo and Nixon proposed a feature selection method based on maximizing
I
(
x
;
Y
), the multidimensional mutual information between feature vector
x
and class variable
Y
. Because computing
I
(
x
;
Y
) can be difficult in practice, Guo and Nixon proposed an approximation of
I
(
x
;
Y
) as the criterion for feature selection. We show that Guo and Nixon's criterion originates from approximating the joint probability distributions in
I
(
x
;
Y
) by second-order product distributions. We remark on the limitations of the approximation and discuss computationally attractive alternatives to compute
I
(
x
;
Y
) .