REU: AitF: Mechanism Design and Machine Learning for Peer Grading

The classroom is an outstanding environment in which to develop and understand algorithms beyond digital computers. Whereas traditional algorithms map inputs to outputs via an exactly specified process, algorithms beyond digital computers must solicit inputs which can be unreliable or strategic and rely on computational building blocks that may be unreliable or strategic. The classroom gives an isolated environment in which to design, analyze, and test these new kinds of algorithms. Moreover, the exploding enrollments in computer science classes globally requires greater optimization of the use of limited teaching resources, in particular in grading student work. Thus, the classroom provides a unique opportunity to bring the theory of algorithms to practice and satisfy a fundamental societal need. We propose to design, build, and deploy a system for peer grading of homework assignments in large lecture classes to reduce the grading load on course staff. In such a system students in the class would be solicited for peer reviews of homework submissions. The peer grading system will aggregate multiple peer reviews to determine the grade on the homework submissions. The system will grade the peer reviews for their accuracy. A student’s grade on the homework exercise will result in part from his/her ability to find his/her own solution and in part from his/her ability to understand whether other students' solutions are correct. The system can make use of course staff to provide the ground-truth grade for any homework submission; however, one system goal is to minimize the grading load of course staff. Keywords: peer review, mechanism design, machine learning. Intellectual Merit. This proposal adapts theoretical methods from algorithmic game theory to design and analyze peer grading systems and directly evaluates the resulting theory with outcomes from a peer review system that the PIs will build. The main concern of this proposal is understanding how theory maps on to practice, identifying which theories are practical, and developing new theories where existing theories are impractical. Broader Impact. In CS classrooms there is a significant need for more efficient use of TA resources, a need that the peer review systems the PIs propose to build will address. Moreover, peer review systems provide a strongly motivated medium to interest students the theoretical foundations of peer review systems, namely, algorithms, game theory, mechanism design, and machine learning. Beyond CS classrooms, a successful peer review mechanism could have impact on reducing the cost of education broadly, and enable a refocusing of teacher efforts on curriculum development and teaching rather than grading.