TY - GEN

T1 - Relative entropy and score function

T2 - 2009 IEEE International Symposium on Information Theory, ISIT 2009

AU - Guo, Dongning

PY - 2009

Y1 - 2009

N2 - This paper establishes new information-estimation relationships pertaining to models with additive noise of arbitrary distribution. In particular, we study the change in the relative entropy between two probability measures when both of them are perturbed by a small amount of the same additive noise. It is shown that the rate of the change with respect to the energy of the perturbation can be expressed in terms of the mean squared difference of the score functions of the two distributions, and, rather surprisingly, is unrelated to the distribution of the perturbation otherwise. The result holds true for the classical relative entropy (or Kullback-Leibler distance), as well as two of its generalizations: Rényi's relative entropy and the f-divergence. The result generalizes a recent relationship between the relative entropy and mean squared errors pertaining to Gaussian noise models, which in turn supersedes many previous information- estimation relationships. A generalization of the de Bruijn identity to non-Gaussian models can also be regarded as consequence of this new result.

AB - This paper establishes new information-estimation relationships pertaining to models with additive noise of arbitrary distribution. In particular, we study the change in the relative entropy between two probability measures when both of them are perturbed by a small amount of the same additive noise. It is shown that the rate of the change with respect to the energy of the perturbation can be expressed in terms of the mean squared difference of the score functions of the two distributions, and, rather surprisingly, is unrelated to the distribution of the perturbation otherwise. The result holds true for the classical relative entropy (or Kullback-Leibler distance), as well as two of its generalizations: Rényi's relative entropy and the f-divergence. The result generalizes a recent relationship between the relative entropy and mean squared errors pertaining to Gaussian noise models, which in turn supersedes many previous information- estimation relationships. A generalization of the de Bruijn identity to non-Gaussian models can also be regarded as consequence of this new result.

UR - http://www.scopus.com/inward/record.url?scp=70449469739&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=70449469739&partnerID=8YFLogxK

U2 - 10.1109/ISIT.2009.5205652

DO - 10.1109/ISIT.2009.5205652

M3 - Conference contribution

AN - SCOPUS:70449469739

SN - 9781424443130

T3 - IEEE International Symposium on Information Theory - Proceedings

SP - 814

EP - 818

BT - 2009 IEEE International Symposium on Information Theory, ISIT 2009

Y2 - 28 June 2009 through 3 July 2009

ER -