Using large corpus N-gram statistics to improve recurrent neural language models

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Recurrent neural network language models (RNNLM) form a valuable foundation for many NLP systems, but training the models can be computationally expensive, and may take days to train on a large corpus. We explore a technique that uses large corpus n-gram statistics as a regularizer for training a neural network LM on a smaller corpus. In experiments with the Billion-Word and Wikitext corpora, we show that the technique is effective, and more time-efficient than simply training on a larger sequential corpus. We also introduce new strategies for selecting the most informative n-grams, and show that these boost efficiency.

Original languageEnglish (US)
Title of host publicationLong and Short Papers
PublisherAssociation for Computational Linguistics (ACL)
Pages3268-3273
Number of pages6
ISBN (Electronic)9781950737130
StatePublished - 2019
Event2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2019 - Minneapolis, United States
Duration: Jun 2 2019Jun 7 2019

Publication series

NameNAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference
Volume1

Conference

Conference2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2019
CountryUnited States
CityMinneapolis
Period6/2/196/7/19

ASJC Scopus subject areas

  • Language and Linguistics
  • Computer Science Applications
  • Linguistics and Language

Fingerprint Dive into the research topics of 'Using large corpus N-gram statistics to improve recurrent neural language models'. Together they form a unique fingerprint.

  • Cite this

    Yang, Y., Wang, J. P., & Downey, D. (2019). Using large corpus N-gram statistics to improve recurrent neural language models. In Long and Short Papers (pp. 3268-3273). (NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference; Vol. 1). Association for Computational Linguistics (ACL).