Turkers, Scholars, "arafat" and "peace": Cultural communities and algorithmic gold standards

Shilad Sen, Margaret E. Giesel, Rebecca Gold, Benjamin Hillmann, Matt Lesicko, Samuel Naden, Jesse Russell, Zixiao Ken Wang, Brent Hecht

Research output: Chapter in Book/Report/Conference proceedingConference contribution

22 Scopus citations

Abstract

In just a few years, crowdsourcing markets like Mechanical Turk have become the dominant mechanism for for building "gold standard" datasets in areas of computer science ranging from natural language processing to audio transcription. The assumption behind this sea change-An assumption that is central to the approaches taken in hundreds of research projects-is that crowdsourced markets can accurately replicate the judgments of the general population for knowledgeoriented tasks. Focusing on the important domain of semantic relatedness algorithms and leveraging Clark's theory of common ground as a framework, we demonstrate that this assumption can be highly problematic. Using 7,921 semantic relatedness judgements from 72 scholars and 39 crowdworkers, we show that crowdworkers on Mechanical Turk produce significantly different semantic relatedness gold standard judgements than people from other communities. We also show that algorithms that perform well against Mechanical Turk gold standard datasets do significantly worse when evaluated against other communities' gold standards. Our results call into question the broad use of Mechanical Turk for the development of gold standard datasets and demonstrate the importance of understanding these datasets from a human-centered point-of-view. More generally, our findings problematize the notion that a universal gold standard dataset exists for all knowledge tasks.

Original languageEnglish (US)
Title of host publicationCSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing
PublisherAssociation for Computing Machinery, Inc
Pages826-838
Number of pages13
ISBN (Electronic)9781450329224
DOIs
StatePublished - Feb 28 2015
Event18th ACM International Conference on Computer-Supported Cooperative Work and Social Computing, CSCW 2015 - BC, Canada
Duration: Mar 14 2015Mar 18 2015

Publication series

NameCSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing

Other

Other18th ACM International Conference on Computer-Supported Cooperative Work and Social Computing, CSCW 2015
CountryCanada
CityBC
Period3/14/153/18/15

Keywords

  • Amazon Mechanical Turk
  • cultural communities
  • gold standard datasets
  • natural language processing
  • semantic relatedness
  • user studies

ASJC Scopus subject areas

  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'Turkers, Scholars, "arafat" and "peace": Cultural communities and algorithmic gold standards'. Together they form a unique fingerprint.

  • Cite this

    Sen, S., Giesel, M. E., Gold, R., Hillmann, B., Lesicko, M., Naden, S., Russell, J., Wang, Z. K., & Hecht, B. (2015). Turkers, Scholars, "arafat" and "peace": Cultural communities and algorithmic gold standards. In CSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing (pp. 826-838). (CSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing). Association for Computing Machinery, Inc. https://doi.org/10.1145/2675133.2675285