Evaluating the portfolio's ability to identify behavioral patterns in medical students that impact patient care

The Feinberg School of Medicine at Northwestern University has added a portfolio-based assessment system to supplement its graded curriculum in hopes of measuring and developing behavioral competencies such as communication, professionalism and teamwork in students. The purpose of this study is to empirically evaluate the ability of the electronic portfolio review process (explained below) to identify behavioral patterns in the pre-clerkship phase that persist into clerkships, potentially influencing performance. We seek to address the following question: What is the relationship between student performance in behavioral competencies in the pre-clerkship phase compared to clerkships, as reflected in narrative feedback collected in the portfolio? The results of this project will help to identify themes in behavior, allowing us to target interventions to help struggling students earlier in their education. Validating the pre-clerkship portfolio committee’s findings may also result in students’ commitment to improving behavior they may not recognize as problematic. The portfolio is a repository of student assessment data across multiple courses and clerkships, including quantitative ratings and narrative feedback (e.g. comments) from both faculty and peers. At the end of the pre-clerkship phase of our curriculum, a faculty committee reviews portfolios to detect themes in performance and identify students who would benefit from remediation in one or more behavioral competencies before starting clerkships. These reviews resulted in summative decisions about performance. We ran quantitative analyses to explore the relationship of the pre-clerkship portfolio decisions with clerkship grades. Preliminary results of the multiple regression show that concerning behavior identified by the pre-clerkship portfolio review significantly and negatively predicted clerkship grades, even after controlling for academic ability as measured by USMLE Step 1 score (R2=.50, p<.001). These results suggest that the portfolio review is identifying behaviors that persist into clerkships, but statistics alone cannot fully explain this process. Only qualitative analyses of these results will allow us to understand whether the portfolio can identify behaviors that impact clinical performance in the clerkship year. We will utilize qualitative methodology to code and analyze the narrative feedback from both the portfolio reviewers and clerkship directors. Because the portfolio contains a large amount of this feedback, it has been difficult to devote the necessary time and resources to carry the project to completion. This proposal requests funding for a part-time research assistant to help analyze the qualitative assessment data.