A note of caution on maximizing entropy

Richard E. Neapolitan*, Xia Jiang

*Corresponding author for this work

    Research output: Contribution to journalArticlepeer-review

    2 Scopus citations

    Abstract

    The Principle of Maximum Entropy is often used to update probabilities due to evidence instead of performing Bayesian updating using Bayes' Theorem, and its use often has efficacious results. However, in some circumstances the results seem unacceptable and unintuitive. This paper discusses some of these cases, and discusses how to identify some of the situations in which this principle should not be used. The paper starts by reviewing three approaches to probability, namely the classical approach, the limiting frequency approach, and the Bayesian approach. It then introduces maximum entropy and shows its relationship to the three approaches. Next, through examples, it shows that maximizing entropy sometimes can stand in direct opposition to Bayesian updating based on reasonable prior beliefs. The paper concludes that if we take the Bayesian approach that probability is about reasonable belief based on all available information, then we can resolve the conflict between the maximum entropy approach and the Bayesian approach that is demonstrated in the examples.

    Original languageEnglish (US)
    Pages (from-to)4004-4014
    Number of pages11
    JournalEntropy
    Volume16
    Issue number7
    DOIs
    StatePublished - 2014

    Keywords

    • Bayesian
    • Classical approach
    • Limiting frequency
    • Maximum entropy
    • Subjective probability

    ASJC Scopus subject areas

    • Information Systems
    • Mathematical Physics
    • Physics and Astronomy (miscellaneous)
    • General Physics and Astronomy
    • Electrical and Electronic Engineering

    Fingerprint

    Dive into the research topics of 'A note of caution on maximizing entropy'. Together they form a unique fingerprint.

    Cite this