Sampling informative training data for RNN language models

Jared Fernandez; Douglas C Downey

doi:10.18653/v1/p18-3002

Sampling informative training data for RNN language models

Jared Fernandez^*, Douglas C Downey

^*Corresponding author for this work

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

7 Scopus citations

Abstract

We propose an unsupervised importance sampling approach to selecting training data for recurrent neural network (RNN) language models. To increase the information content of the training set, our approach preferentially samples high perplexity sentences, as determined by an easily queryable n-gram language model. We experimentally evaluate the heldout perplexity of models trained with our various importance sampling distributions. We show that language models trained on data sampled using our proposed approach outperform models trained over randomly sampled subsets of both the Billion Word (Chelba et al., 2014) and Wikitext-103 benchmark corpora (Merity et al., 2016).

Original language	English (US)
Title of host publication	ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Student Research Workshop
Publisher	Association for Computational Linguistics (ACL)
Pages	9-13
Number of pages	5
ISBN (Electronic)	9781948087360
DOIs	https://doi.org/10.18653/v1/p18-3002
State	Published - 2018
Event	56th Annual Meeting of the Association for Computational Linguistics, ACL 2018 - Melbourne, Australia Duration: Jul 15 2018 → Jul 20 2018

Publication series

Name	ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Student Research Workshop

Conference

Conference	56th Annual Meeting of the Association for Computational Linguistics, ACL 2018
Country/Territory	Australia
City	Melbourne
Period	7/15/18 → 7/20/18

ASJC Scopus subject areas

Software
Computational Theory and Mathematics

Access to Document

10.18653/v1/p18-3002

Cite this

Fernandez, J., & Downey, D. C. (2018). Sampling informative training data for RNN language models. In ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Student Research Workshop (pp. 9-13). (ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Student Research Workshop). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/p18-3002

Fernandez, Jared ; Downey, Douglas C. / Sampling informative training data for RNN language models. ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Student Research Workshop. Association for Computational Linguistics (ACL), 2018. pp. 9-13 (ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Student Research Workshop).

@inproceedings{57316b6cf370459495d33db67efba958,

title = "Sampling informative training data for RNN language models",

abstract = "We propose an unsupervised importance sampling approach to selecting training data for recurrent neural network (RNN) language models. To increase the information content of the training set, our approach preferentially samples high perplexity sentences, as determined by an easily queryable n-gram language model. We experimentally evaluate the heldout perplexity of models trained with our various importance sampling distributions. We show that language models trained on data sampled using our proposed approach outperform models trained over randomly sampled subsets of both the Billion Word (Chelba et al., 2014) and Wikitext-103 benchmark corpora (Merity et al., 2016).",

author = "Jared Fernandez and Downey, {Douglas C}",

note = "Publisher Copyright: {\textcopyright} 2018 Association for Computational Linguistics.; 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018 ; Conference date: 15-07-2018 Through 20-07-2018",

year = "2018",

doi = "10.18653/v1/p18-3002",

language = "English (US)",

series = "ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Student Research Workshop",

publisher = "Association for Computational Linguistics (ACL)",

pages = "9--13",

booktitle = "ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Student Research Workshop",

}

Fernandez, J & Downey, DC 2018, Sampling informative training data for RNN language models. in ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Student Research Workshop. ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Student Research Workshop, Association for Computational Linguistics (ACL), pp. 9-13, 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, 7/15/18. https://doi.org/10.18653/v1/p18-3002

Sampling informative training data for RNN language models. / Fernandez, Jared; Downey, Douglas C.
ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Student Research Workshop. Association for Computational Linguistics (ACL), 2018. p. 9-13 (ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Student Research Workshop).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Sampling informative training data for RNN language models

AU - Fernandez, Jared

AU - Downey, Douglas C

PY - 2018

Y1 - 2018

N2 - We propose an unsupervised importance sampling approach to selecting training data for recurrent neural network (RNN) language models. To increase the information content of the training set, our approach preferentially samples high perplexity sentences, as determined by an easily queryable n-gram language model. We experimentally evaluate the heldout perplexity of models trained with our various importance sampling distributions. We show that language models trained on data sampled using our proposed approach outperform models trained over randomly sampled subsets of both the Billion Word (Chelba et al., 2014) and Wikitext-103 benchmark corpora (Merity et al., 2016).

AB - We propose an unsupervised importance sampling approach to selecting training data for recurrent neural network (RNN) language models. To increase the information content of the training set, our approach preferentially samples high perplexity sentences, as determined by an easily queryable n-gram language model. We experimentally evaluate the heldout perplexity of models trained with our various importance sampling distributions. We show that language models trained on data sampled using our proposed approach outperform models trained over randomly sampled subsets of both the Billion Word (Chelba et al., 2014) and Wikitext-103 benchmark corpora (Merity et al., 2016).

UR - http://www.scopus.com/inward/record.url?scp=85063074407&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85063074407&partnerID=8YFLogxK

U2 - 10.18653/v1/p18-3002

DO - 10.18653/v1/p18-3002

M3 - Conference contribution

AN - SCOPUS:85063074407

T3 - ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Student Research Workshop

SP - 9

EP - 13

BT - ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Student Research Workshop

PB - Association for Computational Linguistics (ACL)

T2 - 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018

Y2 - 15 July 2018 through 20 July 2018

ER -

Fernandez J, Downey DC. Sampling informative training data for RNN language models. In ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Student Research Workshop. Association for Computational Linguistics (ACL). 2018. p. 9-13. (ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Student Research Workshop). doi: 10.18653/v1/p18-3002

Sampling informative training data for RNN language models

Abstract

Publication series

Conference

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this