Abstract
In this study, we explored the potential for machine scoring of short written responses to the Classroom-Video-Analysis (CVA) assessment, which is designed to measure teachers’ usable mathematics teaching knowledge. We created naïve Bayes classifiers for CVA scales assessing three different topic areas and compared computer-generated scores to those assigned by trained raters. Using cross-validation techniques, average correlations between rater- and computer-generated total scores exceeded.85 for each assessment, providing some evidence for convergent validity of machine scores. These correlations remained moderate to large when we controlled for length of response. Machine scores exhibited internal consistency, which we view as a measure of reliability. Finally, correlations between machine scores and another measure of teacher knowledge were close in size to those observed for human scores, providing further evidence for the validity of machine scores. Findings from this study suggest that machine learning techniques hold promise for automating scoring of the CVA.
Original language | English (US) |
---|---|
Pages (from-to) | 950-974 |
Number of pages | 25 |
Journal | Educational and Psychological Measurement |
Volume | 74 |
Issue number | 6 |
DOIs | |
State | Published - Dec 24 2014 |
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Work on this study was supported by the Institute of Education Sciences, Grant No. R305M060007.
Keywords
- automated scoring
- classroom-video-analysis assessment
- naïve Bayes
- short answer items
- teacher knowledge
- text classification
ASJC Scopus subject areas
- Education
- Developmental and Educational Psychology
- Applied Psychology
- Applied Mathematics