Teaching citizen scientists to categorize glitches using machine learning guided training

Corey Jackson*, Carsten Østerlund, Kevin Crowston, Mahboobeh Harandi, Sarah Allen, Sara Bahaadini, Scotty Coughlin, Vicky Kalogera, Aggelos Katsaggelos, Shane Larson, Neda Rohani, Joshua Smith, Laura Trouille, Michael Zevin

*Corresponding author for this work

Research output: Contribution to journalArticle

Abstract

Existing literature points to scaffolded training as an effective yet resource-intensive approach to help newcomers learn and stay motivated. Experts need to select relevant learning materials and continuously assess learners' progress. Peer production communities such as Wikipedia and Open Source Software Development projects face the additional problem of turning volunteers into productive participants as soon as possible. To address these challenges, we designed and tested a training regime combining scaffolded instruction and machine learning to select learning materials and gradually introduces new materials to individuals as their competences improve. We evaluated the training regime on 386 participants that contribute to Gravity Spy, an online citizen science project where people are asked to categorize glitches to assist scientists in the search for gravitational waves. Volunteers were assigned to one of two conditions; (1) a machine learning guided training (MLGT) system that continuously assesses volunteers skill level and adjusts the learning materials or (2) an unscaffolded training program where all learning materials were administered at once. Our analysis revealed that volunteers in the MLGT condition were more accurate on the categorization task (an average accuracy of 90% vs. 54%), executed more tasks (an average of 228 vs. 121 tasks), and were retained for a longer period (an average of 2.5 vs. 2 sessions) than volunteers in the unscaffolded training. The results suggest that MLGT is an effective pedagogical approach for training volunteers in categorization tasks and increases volunteers’ motivation.

Original languageEnglish (US)
Article number106198
JournalComputers in Human Behavior
Volume105
DOIs
StatePublished - Apr 2020

    Fingerprint

Keywords

  • Citizen science
  • Experiment
  • Learning
  • Online communities
  • Scaffolding
  • Training
  • User studies
  • Zooniverse

ASJC Scopus subject areas

  • Arts and Humanities (miscellaneous)
  • Human-Computer Interaction
  • Psychology(all)

Cite this

Jackson, C., Østerlund, C., Crowston, K., Harandi, M., Allen, S., Bahaadini, S., Coughlin, S., Kalogera, V., Katsaggelos, A., Larson, S., Rohani, N., Smith, J., Trouille, L., & Zevin, M. (2020). Teaching citizen scientists to categorize glitches using machine learning guided training. Computers in Human Behavior, 105, [106198]. https://doi.org/10.1016/j.chb.2019.106198