Learning From Crowds by Class-Specific Instance Weighting

In crowdsourcing scenarios, we can acquire each instance's multiple noisy label set from crowd workers and then infer its integrated label via label integration. To further enhance the quality of integrated labels, some noise correction algorithms have been proposed in recent years. Most of them aim to partition the dataset into a clean set and a noise set, followed by training one or multiple classifiers on the clean set to rectify the instances in the noise set. However, they overlook the fact that the class distribution of the clean set is often inconsistent with that of the noise set, resulting in the subpar correction performance of trained classifiers. To mitigate this inconsistency, this paper proposes a class-specific instance weighting-based noise correction (CIWNC) algorithm. In CIWNC, each class's weight is computed based on the class distribution of the clean set firstly. Subsequently, a class-specific weight is computed for each instance using the weight of the class that its integrated label belongs to, as well as its multiple noisy label set. Finally, a classifier is trained on the instance weighted clean set to rectify the instances in the noise set. Experimental results on 34 simulated and two real-world datasets demonstrate that CIWNC outperforms existing state-of-the-art noise correction algorithms significantly.