TY - JOUR
T1 - Designing Crowdcritique Systems for Formative Feedback
AU - Easterday, Matthew W.
AU - Rees Lewis, Daniel
AU - Gerber, Elizabeth M.
N1 - Funding Information:
This work supported by the National Science Foundation Grant No. IIS-1320693, No. IIS-1530833, and No. IIS-1217225 and Venture Well. An earlier version of study 3 was reported at the Proceedings of the 2014 Designing Interactive Systems Conference.
Publisher Copyright:
© 2016, International Artificial Intelligence in Education Society.
PY - 2017/9/1
Y1 - 2017/9/1
N2 - Intelligent tutors based on expert systems often struggle to provide formative feedback on complex, ill-defined problems where answers are unknown. Hybrid crowdsourcing systems that combine the intelligence of multiple novices in face-to-face settings might provide an alternate approach for providing intelligent formative feedback. The purpose of this study was to develop empirically grounded design principles for crowdcritique systems that provide intelligent formative feedback on complex, ill-defined problems. In this design research project, we iteratively developed and tested a crowdcritique system through 3 studies of 43 novice problem solvers in 3 formal and informal learning environments. We collected observations, interviews, and surveys and used a grounded theory approach to develop and test socio-technical design principles for crowdcritique systems. The project found that to provide formative feedback on ill-defined problems, crowdcritique systems should provide a combination of technical features including: quick invite tools; formative framing; a public, near-synchronous social media interface; critique scaffolds; “like” system; community hashtags; analysis tools and “to do” lists; along with social practices including: prep/write-first/write-last script and critique training. Such a system creates a dual-channel conversation that increases the volume of quality critique by grounding comments, scaffolding and recording critique, and reducing production blocking. Such a design provides the benefits of both face-to-face critique and computer-support in both formal and informal learning environments while reducing the orchestration burden on instructors.
AB - Intelligent tutors based on expert systems often struggle to provide formative feedback on complex, ill-defined problems where answers are unknown. Hybrid crowdsourcing systems that combine the intelligence of multiple novices in face-to-face settings might provide an alternate approach for providing intelligent formative feedback. The purpose of this study was to develop empirically grounded design principles for crowdcritique systems that provide intelligent formative feedback on complex, ill-defined problems. In this design research project, we iteratively developed and tested a crowdcritique system through 3 studies of 43 novice problem solvers in 3 formal and informal learning environments. We collected observations, interviews, and surveys and used a grounded theory approach to develop and test socio-technical design principles for crowdcritique systems. The project found that to provide formative feedback on ill-defined problems, crowdcritique systems should provide a combination of technical features including: quick invite tools; formative framing; a public, near-synchronous social media interface; critique scaffolds; “like” system; community hashtags; analysis tools and “to do” lists; along with social practices including: prep/write-first/write-last script and critique training. Such a system creates a dual-channel conversation that increases the volume of quality critique by grounding comments, scaffolding and recording critique, and reducing production blocking. Such a design provides the benefits of both face-to-face critique and computer-support in both formal and informal learning environments while reducing the orchestration burden on instructors.
KW - Complex problems
KW - Critique
KW - Crowdsourcing
KW - Crowdwork
KW - Design principles
KW - Formative feedback
KW - Project-based learning
KW - Social-technical systems
UR - http://www.scopus.com/inward/record.url?scp=85025069894&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85025069894&partnerID=8YFLogxK
U2 - 10.1007/s40593-016-0125-9
DO - 10.1007/s40593-016-0125-9
M3 - Article
AN - SCOPUS:85025069894
VL - 27
SP - 623
EP - 663
JO - International Journal of Artificial Intelligence in Education
JF - International Journal of Artificial Intelligence in Education
SN - 1560-4292
IS - 3
ER -