Four Types of Learning Curves

If machines are learning to make decisions given a number of examples, the generalization error ϵ(t) is defined as the average probability that an incorrect decision is made for a new example by a machine when trained with t examples. The generalization error decreases as t increases, and the curve ϵ(t) is called a learning curve. The present paper uses the Bayesian approach to show that given the annealed approximation, learning curves can be classified into four asymptotic types. If the machine is deterministic with noiseless teacher signals, then (1) ϵ ∼ at^-1 when the correct machine parameter is unique, and (2) ϵ ∼ at^-2 when the set of the correct parameters has a finite measure. If the teacher signals are noisy, then (3) ϵ ∼ at^-1/2 for a deterministic machine, and (4) ϵ ∼ c + at^-1 for a stochastic machine.