Choosing an appropriate model for novelty detection

For novelty detection, a description of normality is learnt by fitting a model to the set of normal examples, and previously unseen patterns are then tested by comparing their novelty score (as defined by the model) against some threshold. Models need to be assessed not only according to their ability to separate normal and abnormal examples but also according to the extent of overfitting relative to the training set. When the amount of training data is small, the effects of overfitting can be estimated by using cross-validation to determine how many previously unseen normal patterns would be classified correctly. This is illustrated by considering the problem of identifying unusual jet engine vibration signatures. With a larger training database, a principled approach can be adopted which provides both a measure of the quality of the model and a means of determining the value of the novelty threshold.