Abstract
Neural Network Language Models (NNLMs) generate probability distributions by applying a softmax function to a distance metric formed by taking the dot product of a prediction vector with all word vectors in a high-dimensional embedding space. The dot-product distance metric forms part of the inductive bias of NNLMs. Although NNLMs optimize well with this inductive bias, we show that this results in a sub-optimal ordering of the embedding space that structurally impoverishes some words at the expense of others when assigning probability. We present numerical, theoretical and empirical analyses showing that words on the interior of the convex hull in the embedding space have their probability bounded by the probabilities of the words on the hull.
| Original language | English (US) |
|---|---|
| Title of host publication | ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference |
| Publisher | Association for Computational Linguistics (ACL) |
| Pages | 2191-2197 |
| Number of pages | 7 |
| ISBN (Electronic) | 9781952148255 |
| DOIs | |
| State | Published - 2020 |
| Event | 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020 - Virtual, Online, United States Duration: Jul 5 2020 → Jul 10 2020 |
Publication series
| Name | Proceedings of the Annual Meeting of the Association for Computational Linguistics |
|---|---|
| ISSN (Print) | 0736-587X |
Conference
| Conference | 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020 |
|---|---|
| Country/Territory | United States |
| City | Virtual, Online |
| Period | 7/5/20 → 7/10/20 |
Funding
This work was supported in part by NSF Grant IIS-1351029. We thank the anonymous reviewers and Northwestern\u2019s Theoretical Computer Science group for their insightful comments and guidance.
ASJC Scopus subject areas
- Computer Science Applications
- Linguistics and Language
- Language and Linguistics