Designing Crowdcritique Systems for Formative Feedback

Matthew W. Easterday*, Daniel Rees Lewis, Elizabeth M. Gerber

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

9 Scopus citations


Intelligent tutors based on expert systems often struggle to provide formative feedback on complex, ill-defined problems where answers are unknown. Hybrid crowdsourcing systems that combine the intelligence of multiple novices in face-to-face settings might provide an alternate approach for providing intelligent formative feedback. The purpose of this study was to develop empirically grounded design principles for crowdcritique systems that provide intelligent formative feedback on complex, ill-defined problems. In this design research project, we iteratively developed and tested a crowdcritique system through 3 studies of 43 novice problem solvers in 3 formal and informal learning environments. We collected observations, interviews, and surveys and used a grounded theory approach to develop and test socio-technical design principles for crowdcritique systems. The project found that to provide formative feedback on ill-defined problems, crowdcritique systems should provide a combination of technical features including: quick invite tools; formative framing; a public, near-synchronous social media interface; critique scaffolds; “like” system; community hashtags; analysis tools and “to do” lists; along with social practices including: prep/write-first/write-last script and critique training. Such a system creates a dual-channel conversation that increases the volume of quality critique by grounding comments, scaffolding and recording critique, and reducing production blocking. Such a design provides the benefits of both face-to-face critique and computer-support in both formal and informal learning environments while reducing the orchestration burden on instructors.

Original languageEnglish (US)
Pages (from-to)623-663
Number of pages41
JournalInternational Journal of Artificial Intelligence in Education
Issue number3
StatePublished - Sep 1 2017


  • Complex problems
  • Critique
  • Crowdsourcing
  • Crowdwork
  • Design principles
  • Formative feedback
  • Project-based learning
  • Social-technical systems

ASJC Scopus subject areas

  • Education
  • Computational Theory and Mathematics


Dive into the research topics of 'Designing Crowdcritique Systems for Formative Feedback'. Together they form a unique fingerprint.

Cite this