Toward automatic task design: A progress report

Eric Huang*, Haoqi Zhang, David C. Parkes, Krzysztof Z. Gajos, Yiling Chen

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

46 Scopus citations

Abstract

A central challenge in human computation is in understanding how to design task environments that effectively attract participants and coordinate the problem solving process. In this paper, we consider a common problem that requesters face on Amazon Mechanical Turk: how should a task be designed so as to induce good output from workers? In posting a task, a requester decides how to break down the task into unit tasks, how much to pay for each unit task, and how many workers to assign to a unit task. These design decisions affect the rate at which workers complete unit tasks, as well as the quality of the work that results. Using image labeling as an example task, we consider the problem of designing the task to maximize the number of quality tags received within given time and budget constraints. We consider two different measures of work quality, and construct models for predicting the rate and quality of work based on observations of output to various designs. Preliminary results show that simple models can accurately predict the quality of output per unit task, but are less accurate in predicting the rate at which unit tasks complete. At a fixed rate of pay, our models generate different designs depending on the quality metric, and optimized designs obtain significantly more quality tags than baseline comparisons.

Original languageEnglish (US)
Title of host publicationWorkshop Proceedings - Human Computation Workshop 2010, HCOMP2010
Pages77-85
Number of pages9
DOIs
StatePublished - Sep 8 2010
EventHuman Computation Workshop 2010, HCOMP2010 - Washington, DC, United States
Duration: Jul 25 2010Jul 25 2010

Publication series

NameWorkshop Proceedings - Human Computation Workshop 2010, HCOMP2010

Other

OtherHuman Computation Workshop 2010, HCOMP2010
CountryUnited States
CityWashington, DC
Period7/25/107/25/10

Keywords

  • Human computation
  • Mechanical Turk
  • Peer production

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Human-Computer Interaction

Fingerprint Dive into the research topics of 'Toward automatic task design: A progress report'. Together they form a unique fingerprint.

  • Cite this

    Huang, E., Zhang, H., Parkes, D. C., Gajos, K. Z., & Chen, Y. (2010). Toward automatic task design: A progress report. In Workshop Proceedings - Human Computation Workshop 2010, HCOMP2010 (pp. 77-85). (Workshop Proceedings - Human Computation Workshop 2010, HCOMP2010). https://doi.org/10.1145/1837885.1837908