Language models as representations for weakly-supervised NLP tasks

Fei Huang; Alexander Yates; Arun Ahuja; Douglas C Downey

Language models as representations for weakly-supervised NLP tasks

Fei Huang^*, Alexander Yates, Arun Ahuja, Douglas C Downey

^*Corresponding author for this work

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

9 Scopus citations

Abstract

Finding the right representation for words is critical for building accurate NLP systems when domain-specific labeled data for the task is scarce. This paper investigates language model representations, in which language models trained on unlabeled corpora are used to generate real-valued feature vectors for words. We investigate ngram models and probabilistic graphical models, including a novel lattice-structured Markov Random Field. Experiments indicate that language model representations outperform traditional representations, and that graphical model representations outperform ngram models, especially on sparse and polysemous words.

Original language	English (US)
Title of host publication	CoNLL 2011 - Fifteenth Conference on Computational Natural Language Learning, Proceedings of the Conference
Pages	125-134
Number of pages	10
State	Published - 2011
Event	15th Conference on Computational Natural Language Learning, CoNLL 2011 - Portland, OR, United States Duration: Jun 23 2011 → Jun 24 2011

Publication series

Name	CoNLL 2011 - Fifteenth Conference on Computational Natural Language Learning, Proceedings of the Conference

Other

Other	15th Conference on Computational Natural Language Learning, CoNLL 2011
Country/Territory	United States
City	Portland, OR
Period	6/23/11 → 6/24/11

ASJC Scopus subject areas

Artificial Intelligence
Linguistics and Language
Human-Computer Interaction

Cite this

@inproceedings{aeeab9c6b70e4a6d8b274746a16e2694,

title = "Language models as representations for weakly-supervised NLP tasks",

abstract = "Finding the right representation for words is critical for building accurate NLP systems when domain-specific labeled data for the task is scarce. This paper investigates language model representations, in which language models trained on unlabeled corpora are used to generate real-valued feature vectors for words. We investigate ngram models and probabilistic graphical models, including a novel lattice-structured Markov Random Field. Experiments indicate that language model representations outperform traditional representations, and that graphical model representations outperform ngram models, especially on sparse and polysemous words.",

author = "Fei Huang and Alexander Yates and Arun Ahuja and Downey, {Douglas C}",

year = "2011",

language = "English (US)",

isbn = "9781932432923",

series = "CoNLL 2011 - Fifteenth Conference on Computational Natural Language Learning, Proceedings of the Conference",

pages = "125--134",

booktitle = "CoNLL 2011 - Fifteenth Conference on Computational Natural Language Learning, Proceedings of the Conference",

note = "15th Conference on Computational Natural Language Learning, CoNLL 2011 ; Conference date: 23-06-2011 Through 24-06-2011",

}

Huang, F, Yates, A, Ahuja, A & Downey, DC 2011, Language models as representations for weakly-supervised NLP tasks. in CoNLL 2011 - Fifteenth Conference on Computational Natural Language Learning, Proceedings of the Conference. CoNLL 2011 - Fifteenth Conference on Computational Natural Language Learning, Proceedings of the Conference, pp. 125-134, 15th Conference on Computational Natural Language Learning, CoNLL 2011, Portland, OR, United States, 6/23/11.

Language models as representations for weakly-supervised NLP tasks. / Huang, Fei; Yates, Alexander; Ahuja, Arun et al.
CoNLL 2011 - Fifteenth Conference on Computational Natural Language Learning, Proceedings of the Conference. 2011. p. 125-134 (CoNLL 2011 - Fifteenth Conference on Computational Natural Language Learning, Proceedings of the Conference).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Language models as representations for weakly-supervised NLP tasks

AU - Huang, Fei

AU - Yates, Alexander

AU - Ahuja, Arun

AU - Downey, Douglas C

PY - 2011

Y1 - 2011

N2 - Finding the right representation for words is critical for building accurate NLP systems when domain-specific labeled data for the task is scarce. This paper investigates language model representations, in which language models trained on unlabeled corpora are used to generate real-valued feature vectors for words. We investigate ngram models and probabilistic graphical models, including a novel lattice-structured Markov Random Field. Experiments indicate that language model representations outperform traditional representations, and that graphical model representations outperform ngram models, especially on sparse and polysemous words.

AB - Finding the right representation for words is critical for building accurate NLP systems when domain-specific labeled data for the task is scarce. This paper investigates language model representations, in which language models trained on unlabeled corpora are used to generate real-valued feature vectors for words. We investigate ngram models and probabilistic graphical models, including a novel lattice-structured Markov Random Field. Experiments indicate that language model representations outperform traditional representations, and that graphical model representations outperform ngram models, especially on sparse and polysemous words.

UR - http://www.scopus.com/inward/record.url?scp=84862272580&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84862272580&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:84862272580

SN - 9781932432923

T3 - CoNLL 2011 - Fifteenth Conference on Computational Natural Language Learning, Proceedings of the Conference

SP - 125

EP - 134

BT - CoNLL 2011 - Fifteenth Conference on Computational Natural Language Learning, Proceedings of the Conference

T2 - 15th Conference on Computational Natural Language Learning, CoNLL 2011

Y2 - 23 June 2011 through 24 June 2011

ER -

Language models as representations for weakly-supervised NLP tasks

Abstract

Publication series

Other

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this