Extracting the Pairs of Opinion Target and Opinion Term From Reviews With Adaptive Crowd Labeling

Labeling samples manually is a laborious, error-prone, and cost-consuming process, which is a traditional approach to preparing training data for supervised learning. Crowdsourcing provides an effective way to acquire labeled training data. In this paper, we propose an adaptive crowd labeling method to construct a set of the pairs of opinion target and opinion term iteratively under certain budget constraint. First, we assess the workers’ reliabilities with a small number of labeled samples based on an EM process, since the worker’s expertise varies widely on an open crowdsourcing platform. And then, the tasks are assigned to the high reliable workers for labeling the pairs of opinion target and opinion term. Finally, the responds of a sample from multiple workers are integrated to generate a final result based on the labelers’ reliabilities as well as the dependency relation between opinion target and opinion term in this sample. A series of the experimental results show that the proposed method can achieve better extracting effectiveness compared with the baselines and the state-of-the-art methods.