An automated speaker recognition system for home service robots is proposed in this paper. In an uncontrolled environment, a speech classifier should be adaptive to different users and robust to noisy environments. It is usually observed that specific features and classifiers are more appropriate to parts of the problem domain than others; therefore, we propose a self-optimizing approach in which multiple feature extraction and classification techniques are simultaneously considered. The system uses a genetic algorithm to simultaneously select features and classifier, and the results from multiple classifiers are then combined using the Dempster-Shafer theory. The set of feature extractors used here includes linear-prediction coefficients, linear-prediction cepstral coefficients, mel-frequency cepstral coefficients, and bark-frequency cepstral coefficients, and the set of classifiers includes the Gaussian mixture model, support vector machines, C4.5 decision tree, k nearest neighbors, and multilayer perceptron neural network. The WEVER-R2 home service robot is used in a typical Korean home environment to collect speech signals for evaluating the performance of the proposed system for gender and age classification. Classification results show that the performance of the proposed method consistently outperforms the individual classifiers.