Knowledge acquisition through machine learning: minimising expert's effort

Machine learning can be applied to solve the knowledge acquisition bottleneck in many areas where an expert makes predictions to single cases, such as diagnosis, estimation, etc. The idea is to query the expert with as many cases as possible and get their answers. With this data we train a machine learning model which mimics the expert's behaviour. This is just a simple application of a modelling technique known as "mimetism", which has many other applications. This "soft" approach to knowledge acquisition has many advantages: any machine learning technique can be used, the expert must only answer simple questions (cases) and we can combine the decisions of several experts easily. However, one problem of this approach is that we do not know in advance how many cases we will need to ask in order to get a good model which is accurate wrt. the expert's knowledge. Obviously, as more data is labelled by the expert better results are obtained. However, asking thousands of cases to the expert is usually impractical. In this paper, we analyse the behaviour of knowledge acquisition through mimetic learning according to two factors: accuracy and comprehensibility of the resulting model and we devise a method to compute the minimum number of cases that we need to ask the expert to attain a certain quality level.