Auto-annotation of tomato images based on ripeness and firmness classification for multimodal retrieval

An interesting application of machine vision is quality estimation of tomatoes. Here, there is extensive availability of images that needs to be retrieved in an effective and efficient manner. In this paper, a classification based auto annotation (CBA) of tomato images for providing semantic tags is proposed. These tags are derived from their content-based learning. Semantic tags are deduced from the classification of tomatoes on two most important quality parameters i.e. ripeness and firmness. For firmness estimation, an approach is proposed that exploits three texture feature extraction algorithms: two are based on statistical techniques viz. first order statistics (FOS), and, gray level co-occurrence matrix (GLCM), and one is based on transform-based technique viz. wavelet-transform. Multiple linear regression (MLR) analysis has been used to establish the relationship between the instrumental firmness and texture values obtained from texture analysis. Prediction models for three texture feature sets are built, compared, and tested for over fitting using double cross-validation method. Based on the firmness of tomatoes, which is estimated through digital color imaging technique, they are classified into three classes soft, medium, and hard. To select a classifier for CBA, experiments with five learning methods, Naïve Bayes, multilayer perceptron (MLP), support vector machine (SVM), decision table, and random tree have been carried out. Their class-prediction accuracy has been compared using supplied test set. Ripeness classification of tomato images is done based on color using a previously proposed fuzzy rule based classification (FRBC) approach and are classified into three classes unripe, ab2ripe, and ripe. Grounded on classification, multi-labeling methodology is adopted for automatic annotation of tomato images. From the experimental results it is established that RandomTree classifier performs well for the classification of tomato firmness. In addition, presented work takes the advantage of progression in machine vision to address the issue of semantic gap in multimodal retrieval.