This letter proposes a novel local structure pattern, the generalized binary pattern (GBP), which can represent all possible binary patterns of ordered comparisons within a 3 × 3 neighborhood. Since most existing local structure patterns consider ordered comparisons of all neighboring pixels (8 or 9 pixels) around a given pixel, the number of possible binary patterns is fixed and limited to 256 (or 511). In contrast, our proposed GBP takes the ordered comparisons of some partial neighboring pixels (2 to 9 pixels) around a given pixel, which generates a total of 502 different structure types and thus extends the number of possible binary patterns to 19,162. Among these possible binary patterns, we choose an effective set of binary patterns for a given problem by means of the AdaBoost feature selection method. Experimental results show that our proposed GBP provides higher eye detection accuracy on the BioID and LFW face databases than other existing local structure patterns.