Automated Optimal Online Civil Issue Classification using Multiple Feature Sets

In this paper, the automatic classification of non-emergency civil issues in crowdsourcing systems is addressed in the case where multiple feature sets are available. We recognize that multiple feature sets can contain useful complementary information regarding the type of an issue leading to a more accurate decision. However, using all features in these sets may delay the decision. Since we are interested in reaching an accurate decision in a timely manner, an optimal way of selecting features from multiple feature sets is needed. To this end, we propose a novel approach that sequentially reviews available features and feature sets to decide whether the feature review process must continue in the current set or move to the next one. In the end, when all feature sets have been reviewed, the issue is classified using all available information. It is shown that the proposed approach is guaranteed to review the least number of features in all feature sets before reaching a decision, while the optimum decision rule is shown to minimize the average Bayes risk. Evaluation on real world SeeClickFix data demonstrates the ability to classify issues by reviewing 99.5% less features than state-of-the-art without sacrificing accuracy.