TY - JOUR

T1 - Statistical physics of signal estimation in gaussian noise

T2 - Theory and examples of phase transitions

AU - Merhav, Neri

AU - Guo, Dongning

AU - Shamai, Shlomo

N1 - Funding Information:
Manuscript received December 28, 2008; revised September 25, 2009. Current version published March 10, 2010. The work of D. Guo was supported by the NSF by Grant CCF-0644344 and by DARPA by Grant W911NF-07-1-0028. The work of S. Shamai was supported in part by the Israel Science Foundation. The material in this paper was presented in part at Physics of Algorithms, Santa Fe, NM, August 2009 and in the 2010 Workshop on Information Theory and Applications (ITA), San Diego, CA, February 2010.

PY - 2010/3

Y1 - 2010/3

N2 - We consider the problem of signal estimation (denoising) from a statistical mechanical perspective, using a relationship between the minimum mean square error (MMSE), of estimating a signal, and the mutual information between this signal and its noisy version. The paper consists of essentially two parts. In the first, we derive several statistical-mechanical relationships between a few important quantities in this problem area, such as the MMSE, the differential entropy, the Fisher information, the free energy, and a generalized notion of temperature. We also draw analogies and differences between certain relations pertaining to the estimation problem and the parallel relations in thermodynamics and statistical physics. In the second part of the paper, we provide several application examples, where we demonstrate how certain analysis tools that are customary in statistical physics, prove useful in the analysis of the MMSE. In most of these examples, the corresponding statistical-mechanical systems turn out to consist of strong interactions that cause phase transitions, which in turn are reflected as irregularities and discontinuities (similar to threshold effects) in the behavior of the MMSE.

AB - We consider the problem of signal estimation (denoising) from a statistical mechanical perspective, using a relationship between the minimum mean square error (MMSE), of estimating a signal, and the mutual information between this signal and its noisy version. The paper consists of essentially two parts. In the first, we derive several statistical-mechanical relationships between a few important quantities in this problem area, such as the MMSE, the differential entropy, the Fisher information, the free energy, and a generalized notion of temperature. We also draw analogies and differences between certain relations pertaining to the estimation problem and the parallel relations in thermodynamics and statistical physics. In the second part of the paper, we provide several application examples, where we demonstrate how certain analysis tools that are customary in statistical physics, prove useful in the analysis of the MMSE. In most of these examples, the corresponding statistical-mechanical systems turn out to consist of strong interactions that cause phase transitions, which in turn are reflected as irregularities and discontinuities (similar to threshold effects) in the behavior of the MMSE.

KW - Denoising, de Bruijns identity

KW - Gaussian channel

KW - Minimum mean square error (MMSE) estimation

KW - Phase transitions

KW - Random energy model

KW - Spin glasses

KW - Statistical mechanics

UR - http://www.scopus.com/inward/record.url?scp=77949531339&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=77949531339&partnerID=8YFLogxK

U2 - 10.1109/TIT.2009.2039047

DO - 10.1109/TIT.2009.2039047

M3 - Article

AN - SCOPUS:77949531339

VL - 56

SP - 1400

EP - 1416

JO - IRE Professional Group on Information Theory

JF - IRE Professional Group on Information Theory

SN - 0018-9448

IS - 3

M1 - 5429116

ER -