AitF:Mechanism Design and Machine Learning for Peer Grading

Project: Research project

Project Details


The classroom is an outstanding environment in which to develop and
understand algorithms beyond digital computers. Whereas traditional
algorithms map inputs to outputs via an exactly specified process,
algorithms beyond digital computers must solicit inputs which can be
unreliable or strategic and rely on computational building blocks that
may be unreliable or strategic. The classroom gives an isolated
environment in which to design, analyze, and test these new kinds of
algorithms. Moreover, the exploding enrollments in computer science
classes globally requires greater optimization of the use of limited
teaching resources, in particular in grading student work. Thus, the
classroom provides a unique opportunity to bring the theory of
algorithms to practice and satisfy a fundamental societal need.

We propose to design, build, and deploy a system for peer grading of
homework assignments in large lecture classes to reduce the grading
load on course staff. In such a system students in the class would be
solicited for peer reviews of homework submissions. The peer grading
system will aggregate multiple peer reviews to determine the grade on
the homework submissions. The system will grade the peer reviews for
their accuracy. A student’s grade on the homework exercise will
result in part from his/her ability to find his/her own solution and
in part from his/her ability to understand whether other students'
solutions are correct. The system can make use of course staff to
provide the ground-truth grade for any homework submission; however,
one system goal is to minimize the grading load of course staff.

Keywords: peer review, mechanism design, machine learning.

Intellectual Merit.

This proposal adapts theoretical methods from algorithmic game theory
to design and analyze peer grading systems and directly evaluates the
resulting theory with outcomes from a peer review system that the PIs
will build. The main concern of this proposal is understanding how
theory maps on to practice, identifying which theories are practical,
and developing new theories where existing theories are impractical.

Broader Impact.

In CS classrooms there is a significant need for more efficient use of
TA resources, a need that the peer review systems the PIs propose to
build will address. Moreover, peer review systems provide a strongly
motivated medium to interest students the theoretical foundations of
peer review systems, namely, algorithms, game theory, mechanism
design, and machine learning. Beyond CS classrooms, a successful peer
review mechanism could have impact on reducing the cost of education
broadly, and enable a refocusing of teacher efforts on curriculum
development and teaching rather than grading.
Effective start/end date9/1/178/31/21


  • National Science Foundation (CCF-1733860)


Explore the research topics touched on by this project. These labels are generated based on the underlying awards/grants. Together they form a unique fingerprint.