I don't know the label: Active learning with blind knowledge

Active learning traditionally assumes that the oracle is capable of providing labeling information for each query instance. In reality, the oracle might have no information for some queries and cannot provide accurate label but only answers “I don't know the label”. We focus on this problem and provide a unified objective function to ensure that each query instance submitted to the oracle is the one mostly needed for labeling and the oracle should also have sufficient knowledge to label. Experimental results on real-world and benchmark data sets demonstrate the effectiveness of the proposed design for supporting active learning using oracles with blind knowledge.