Learning in linear neural networks: The validity of the annealed approximation

A statistical-mechanics approach has been recently used to develop a theoretical framework to describe the process of supervised learning and the emergence of generalization ability in layered neural networks. This theory has yielded a powerful tool: a recursion relation that predicts the generalization ability of trained networks as a function of the size of the training set. The recursion relation results from an annealed approximation to the correct quenched averaging over all possible training sets of fixed size. Here we investigate a simple learning problem for which both the quenched and annealed calculations are performed analytically. Comparison between the corresponding results establishes the range of validity of the annealed approximation.