Automated Scoring of Teachers’ Open-Ended Responses to Video Prompts: Bringing the Classroom-Video-Analysis Assessment to Scale

Nicole B. Kersting*, Bruce L. Sherin, James W. Stigler

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

20 Scopus citations

Abstract

In this study, we explored the potential for machine scoring of short written responses to the Classroom-Video-Analysis (CVA) assessment, which is designed to measure teachers’ usable mathematics teaching knowledge. We created naïve Bayes classifiers for CVA scales assessing three different topic areas and compared computer-generated scores to those assigned by trained raters. Using cross-validation techniques, average correlations between rater- and computer-generated total scores exceeded.85 for each assessment, providing some evidence for convergent validity of machine scores. These correlations remained moderate to large when we controlled for length of response. Machine scores exhibited internal consistency, which we view as a measure of reliability. Finally, correlations between machine scores and another measure of teacher knowledge were close in size to those observed for human scores, providing further evidence for the validity of machine scores. Findings from this study suggest that machine learning techniques hold promise for automating scoring of the CVA.

Original languageEnglish (US)
Pages (from-to)950-974
Number of pages25
JournalEducational and Psychological Measurement
Volume74
Issue number6
DOIs
StatePublished - Dec 24 2014

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Work on this study was supported by the Institute of Education Sciences, Grant No. R305M060007.

Keywords

  • automated scoring
  • classroom-video-analysis assessment
  • naïve Bayes
  • short answer items
  • teacher knowledge
  • text classification

ASJC Scopus subject areas

  • Education
  • Developmental and Educational Psychology
  • Applied Psychology
  • Applied Mathematics

Fingerprint

Dive into the research topics of 'Automated Scoring of Teachers’ Open-Ended Responses to Video Prompts: Bringing the Classroom-Video-Analysis Assessment to Scale'. Together they form a unique fingerprint.

Cite this