A Statistical Approach to Learning and Generalization in Layered Neural Networks

Esther Levin; Naftali Tishby; Sara A. Solla

doi:10.1109/5.58339

A Statistical Approach to Learning and Generalization in Layered Neural Networks

Esther Levin, Naftali Tishby, Sara A. Solla

Neuroscience

Research output: Contribution to journal › Article › peer-review

150 Scopus citations

Abstract

A general statistical description of the problem of learning from examples is presented. Our focus is on learning in layered networks, which is posed as a search in the network parameter space for a network that minimizes an additive error function of statistically independent examples. By imposing the equivalence of the minimum error and the maximum likelihood criteria for training the network, we arrive at the Cibbs distribution on the ensemble of networks with a fixed architecture. Using this ensemble, the probability of correct prediction of a novel example can be expressed, serving as a measure of the network’s generalization ability. The entropy of the prediction distribution is shown to be a consistent measure of the network’s performance. This quantity is directly derived from the ensemble statistical properties and is identical to the stochastic complexity of the training data. Our approach is a link between the information-theoretic model-order-estimation techniques, particularly minimum description length, and the statistical mechanics of neural networks. The proposed formalism is applied to the problems of selecting an optimal architecture and the prediction of learning curves.

Original language	English (US)
Pages (from-to)	1568-1574
Number of pages	7
Journal	Proceedings of the IEEE
Volume	78
Issue number	10
DOIs	https://doi.org/10.1109/5.58339
State	Published - Oct 1990

Funding

Naftali 2. Tishby was born in Jerusalem, Israel, in December 1952. He received the B.Sc. degree (cum laude) in physics and mathematics from the Hebrew University of Jerusalem in 1974, M.Sc. degree (cum laude) in physics from Tel-Aviv University in 1980, and the Ph.D. in theoretical physics from Hebrew University in 1985. From 1974 to 1981 he was with the Israel Defense Forces (IDF), where he established and headed a research group in signal and speech processing. During 1984-1985 he served as aVice President of Research in Sesame Systems Ltd. developing speech and speaker recognition systems. In 1985-1986 he was a postdoctoral fellow at the Massachusetts Institute of Technology, working on chaotic Hamiltonian dynamics. Since 1987 he has been a Member of Technical Staff (information principles laboratory) at AT&T Bell Laboratories, Murray Hill, NJ. His current research subjects include nonlinear dynamics and its applications to speech processing, stochastic processes, learning theory, and statistical mechanics of neural networks. Dr. Tishby received the Eliyahu Golomb Israel Security Award in 1980 and the Chaim Weizmann fellowship in physics in 1985.

ASJC Scopus subject areas

General Computer Science
Electrical and Electronic Engineering

Access to Document

10.1109/5.58339

Cite this

@article{4743cc6235384ee3ad0674ba5339d7ec,

title = "A Statistical Approach to Learning and Generalization in Layered Neural Networks",

abstract = "A general statistical description of the problem of learning from examples is presented. Our focus is on learning in layered networks, which is posed as a search in the network parameter space for a network that minimizes an additive error function of statistically independent examples. By imposing the equivalence of the minimum error and the maximum likelihood criteria for training the network, we arrive at the Cibbs distribution on the ensemble of networks with a fixed architecture. Using this ensemble, the probability of correct prediction of a novel example can be expressed, serving as a measure of the network{\textquoteright}s generalization ability. The entropy of the prediction distribution is shown to be a consistent measure of the network{\textquoteright}s performance. This quantity is directly derived from the ensemble statistical properties and is identical to the stochastic complexity of the training data. Our approach is a link between the information-theoretic model-order-estimation techniques, particularly minimum description length, and the statistical mechanics of neural networks. The proposed formalism is applied to the problems of selecting an optimal architecture and the prediction of learning curves.",

author = "Esther Levin and Naftali Tishby and Solla, {Sara A.}",

note = "Funding Information: Naftali 2. Tishby was born in Jerusalem, Israel, in December 1952. He received the B.Sc. degree (cum laude) in physics and mathematics from the Hebrew University of Jerusalem in 1974, M.Sc. degree (cum laude) in physics from Tel-Aviv University in 1980, and the Ph.D. in theoretical physics from Hebrew University in 1985. From 1974 to 1981 he was with the Israel Defense Forces (IDF), where he established and headed a research group in signal and speech processing. During 1984-1985 he served as aVice President of Research in Sesame Systems Ltd. developing speech and speaker recognition systems. In 1985-1986 he was a postdoctoral fellow at the Massachusetts Institute of Technology, working on chaotic Hamiltonian dynamics. Since 1987 he has been a Member of Technical Staff (information principles laboratory) at AT&T Bell Laboratories, Murray Hill, NJ. His current research subjects include nonlinear dynamics and its applications to speech processing, stochastic processes, learning theory, and statistical mechanics of neural networks. Dr. Tishby received the Eliyahu Golomb Israel Security Award in 1980 and the Chaim Weizmann fellowship in physics in 1985.",

year = "1990",

month = oct,

doi = "10.1109/5.58339",

language = "English (US)",

volume = "78",

pages = "1568--1574",

journal = "Proceedings of the IEEE",

issn = "0018-9219",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

number = "10",

}

TY - JOUR

T1 - A Statistical Approach to Learning and Generalization in Layered Neural Networks

AU - Levin, Esther

AU - Tishby, Naftali

AU - Solla, Sara A.

N1 - Funding Information: Naftali 2. Tishby was born in Jerusalem, Israel, in December 1952. He received the B.Sc. degree (cum laude) in physics and mathematics from the Hebrew University of Jerusalem in 1974, M.Sc. degree (cum laude) in physics from Tel-Aviv University in 1980, and the Ph.D. in theoretical physics from Hebrew University in 1985. From 1974 to 1981 he was with the Israel Defense Forces (IDF), where he established and headed a research group in signal and speech processing. During 1984-1985 he served as aVice President of Research in Sesame Systems Ltd. developing speech and speaker recognition systems. In 1985-1986 he was a postdoctoral fellow at the Massachusetts Institute of Technology, working on chaotic Hamiltonian dynamics. Since 1987 he has been a Member of Technical Staff (information principles laboratory) at AT&T Bell Laboratories, Murray Hill, NJ. His current research subjects include nonlinear dynamics and its applications to speech processing, stochastic processes, learning theory, and statistical mechanics of neural networks. Dr. Tishby received the Eliyahu Golomb Israel Security Award in 1980 and the Chaim Weizmann fellowship in physics in 1985.

PY - 1990/10

Y1 - 1990/10

N2 - A general statistical description of the problem of learning from examples is presented. Our focus is on learning in layered networks, which is posed as a search in the network parameter space for a network that minimizes an additive error function of statistically independent examples. By imposing the equivalence of the minimum error and the maximum likelihood criteria for training the network, we arrive at the Cibbs distribution on the ensemble of networks with a fixed architecture. Using this ensemble, the probability of correct prediction of a novel example can be expressed, serving as a measure of the network’s generalization ability. The entropy of the prediction distribution is shown to be a consistent measure of the network’s performance. This quantity is directly derived from the ensemble statistical properties and is identical to the stochastic complexity of the training data. Our approach is a link between the information-theoretic model-order-estimation techniques, particularly minimum description length, and the statistical mechanics of neural networks. The proposed formalism is applied to the problems of selecting an optimal architecture and the prediction of learning curves.

AB - A general statistical description of the problem of learning from examples is presented. Our focus is on learning in layered networks, which is posed as a search in the network parameter space for a network that minimizes an additive error function of statistically independent examples. By imposing the equivalence of the minimum error and the maximum likelihood criteria for training the network, we arrive at the Cibbs distribution on the ensemble of networks with a fixed architecture. Using this ensemble, the probability of correct prediction of a novel example can be expressed, serving as a measure of the network’s generalization ability. The entropy of the prediction distribution is shown to be a consistent measure of the network’s performance. This quantity is directly derived from the ensemble statistical properties and is identical to the stochastic complexity of the training data. Our approach is a link between the information-theoretic model-order-estimation techniques, particularly minimum description length, and the statistical mechanics of neural networks. The proposed formalism is applied to the problems of selecting an optimal architecture and the prediction of learning curves.

UR - http://www.scopus.com/inward/record.url?scp=0025508916&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0025508916&partnerID=8YFLogxK

U2 - 10.1109/5.58339

DO - 10.1109/5.58339

M3 - Article

AN - SCOPUS:0025508916

SN - 0018-9219

VL - 78

SP - 1568

EP - 1574

JO - Proceedings of the IEEE

JF - Proceedings of the IEEE

IS - 10

ER -

A Statistical Approach to Learning and Generalization in Layered Neural Networks

Abstract

Funding

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this