TY - JOUR
T1 - Validation of a Novel Statistical Method to Identify Aberrant Patient Logging
T2 - A Multi-Institutional Study
AU - Min, Elana A.
AU - Lie, Desiree
AU - Barry, Carey
AU - Moloney-Johns, Amanda
AU - Hibbard, Susan T.
AU - Ritsema, Tamara S.
AU - D'Aquila, Mitzi
AU - Honda, Trenton
N1 - Publisher Copyright:
© 2022 Lippincott Williams and Wilkins. All rights reserved.
PY - 2022/9/1
Y1 - 2022/9/1
N2 - Introduction:Student patient encounter logging informs the quality of supervised clinical practice experiences (SCPEs). Yet, it is unknown whether logs accurately reflect patient encounters, and the faculty resources necessary to review for potential aberrant logging are significant. The purpose of this study was to identify a statistical method to identify aberrant logging.Methods:A multi-institutional (n = 6) study examined a statistical method for identifying potentially aberrant logging behavior. An automated statistical Mahalanobis Distance (MD) measurement was used to categorize student logs as aberrant if they were identified as probable multivariate outliers. This approach was validated using a gold standard for aberrant logging behavior with manual review by 4 experienced faculty ("faculty consensus") and then comparing interrater agreement between faculty and MD-based categorization. In secondary analyses, we compared the relative accuracy of MD-based categorization to individual faculty categorizing data from their own program ("own program" categorization).Results:323 student logging records from 6 physician assistant (PA) programs were included. Compared to "faculty consensus" (the gold standard), MD-based categorization was highly sensitive (0.846, 95% CI: 0.650, 1.000) and specific (0.766, 95% CI: 0.645, 0.887). Additionally, there was no significant difference in sensitivity, specificity, positive predictive value, or negative predictive value between MD-based categorization and "own program" categorization.Discussion:The MD-based method of identifying aberrant and nonaberrant student logging compared favorably to the more traditional, faculty-intensive approach of reviewing individual student logging records. This supports MD-based screening as a less labor-intensive alternative to individual faculty review to identify aberrant logging. Identification of aberrant logging may facilitate early intervention with students to improve clinical exposure logging during their SCPEs.
AB - Introduction:Student patient encounter logging informs the quality of supervised clinical practice experiences (SCPEs). Yet, it is unknown whether logs accurately reflect patient encounters, and the faculty resources necessary to review for potential aberrant logging are significant. The purpose of this study was to identify a statistical method to identify aberrant logging.Methods:A multi-institutional (n = 6) study examined a statistical method for identifying potentially aberrant logging behavior. An automated statistical Mahalanobis Distance (MD) measurement was used to categorize student logs as aberrant if they were identified as probable multivariate outliers. This approach was validated using a gold standard for aberrant logging behavior with manual review by 4 experienced faculty ("faculty consensus") and then comparing interrater agreement between faculty and MD-based categorization. In secondary analyses, we compared the relative accuracy of MD-based categorization to individual faculty categorizing data from their own program ("own program" categorization).Results:323 student logging records from 6 physician assistant (PA) programs were included. Compared to "faculty consensus" (the gold standard), MD-based categorization was highly sensitive (0.846, 95% CI: 0.650, 1.000) and specific (0.766, 95% CI: 0.645, 0.887). Additionally, there was no significant difference in sensitivity, specificity, positive predictive value, or negative predictive value between MD-based categorization and "own program" categorization.Discussion:The MD-based method of identifying aberrant and nonaberrant student logging compared favorably to the more traditional, faculty-intensive approach of reviewing individual student logging records. This supports MD-based screening as a less labor-intensive alternative to individual faculty review to identify aberrant logging. Identification of aberrant logging may facilitate early intervention with students to improve clinical exposure logging during their SCPEs.
UR - http://www.scopus.com/inward/record.url?scp=85136630446&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85136630446&partnerID=8YFLogxK
U2 - 10.1097/JPA.0000000000000437
DO - 10.1097/JPA.0000000000000437
M3 - Article
C2 - 35998049
AN - SCOPUS:85136630446
SN - 1941-9430
VL - 33
SP - 192
EP - 197
JO - Journal of Physician Assistant Education
JF - Journal of Physician Assistant Education
IS - 3
ER -