The deterministic information bottleneck

D. J. Strouse, David Jason Schwab

Research output: Chapter in Book/Report/Conference proceedingConference contribution

7 Scopus citations

Abstract

Lossy compression fundamentally involves a decision about what is relevant and what is not. The information bottleneck (IB) by Tishby, Pereira, and Bialek formalized this notion as an information-theoretic optimization problem and proposed an optimal tradeoff between throwing away as many bits as possible, and selectively keeping those that are most important. Here, we introduce an alternative formulation, the deterministic information bottleneck (DIB), that we argue better captures this notion of compression. As suggested by its name, the solution to the DIB problem is a deterministic encoder, as opposed to the stochastic encoder that is optimal under the IB.We then compare the IB and DIB on synthetic data, showing that the IB and DIB perform similarly in terms of the IB cost function, but that the DIB vastly outperforms the IB in terms of the DIB cost function. Moreover, the DIB offered a 1-2 order of magnitude speedup over the IB in our experiments. Our derivation of the DIB also offers a method for continuously interpolating between the soft clustering of the IB and the hard clustering of the DIB.

Original languageEnglish (US)
Title of host publication32nd Conference on Uncertainty in Artificial Intelligence 2016, UAI 2016
EditorsDominik Janzing, Alexander Ihler
PublisherAssociation For Uncertainty in Artificial Intelligence (AUAI)
Pages696-705
Number of pages10
ISBN (Electronic)9781510827806
StatePublished - Jan 1 2016
Event32nd Conference on Uncertainty in Artificial Intelligence 2016, UAI 2016 - Jersey City, United States
Duration: Jun 25 2016Jun 29 2016

Publication series

Name32nd Conference on Uncertainty in Artificial Intelligence 2016, UAI 2016

Other

Other32nd Conference on Uncertainty in Artificial Intelligence 2016, UAI 2016
CountryUnited States
CityJersey City
Period6/25/166/29/16

ASJC Scopus subject areas

  • Artificial Intelligence

Fingerprint Dive into the research topics of 'The deterministic information bottleneck'. Together they form a unique fingerprint.

Cite this