Multicenter Interrater Reliability in the Endoscopic Assessment of Velopharyngeal Function Using a Video Instruction Tool

Craig Miller*, Randall Bly, Shelagh Cofer, John P. Dahl, Lianne de Serres, Steven Goudy, Larry Hartzell, Noel Jabbour, Deborah Kacmarysnki, Carol Macarthur, Anna Messner, Henry Milczuk, Jeff C Rastatter, Laura Sweibel Rosenthal, Andrew Scott, Travis Tollefson, Audie Woolley, Carlton Zdanski, Dave A. Zopf, Kathleen Sie

*Corresponding author for this work

Research output: Contribution to journalArticle

Abstract

Objective: Assess interrater agreement of endoscopic assessment of velopharyngeal (VP) function before and after viewing the video instruction tool (VIT). We hypothesized improvement in interrater agreement using the Golding-Kushner scale (GKS) after viewing the VIT. Study Design: Prospective study. Setting: Multi-institutional. Methods: Sixteen fellowship-trained pediatric otolaryngologists who treat velopharyngeal insufficiency (VPI) rated 50 video segments using the GKS before and after watching the VIT. Raters assessed gap size percentage and lateral pharyngeal wall (LPW), soft palate (SP), and posterior pharyngeal wall (PPW) movement. Intraclass correlation coefficient was estimated for these continuous measures. Raters also indicated the presence of a palatal notch, Passavant’s ridge, and aberrant pulsations (categorical variables). Fleiss κ coefficient was used for categorical variables. Wilcoxon signed-rank test was performed on the difference between the pre/post individual video ratings. Results: Reliability improved for all continuous variables after watching the instructional video. The improvement was significant for PPW (0.22-0.30, P <.001), SP (left: 0.63-0.68, P <.001 and right: 0.64-0.68, P =.001), and LPW (left: 0.49-0.54, P =.01 and right: 0.49-0.54, P =.09) but not significant for gap size (0.65-0.69, P =.36). Among categorical variables, agreement on Passavant’s ridge significantly improved (0.30-0.36, P =.03). Conclusion: Exposure to a video instruction tool improves interrater agreement of endoscopic assessment of VP function. Significant improvement was observed in our primary end points, specifically posterior pharyngeal wall movement, soft palate movement, and lateral pharyngeal wall movement. There was less impact of the VIT on the interrater agreement of the categorical variables, palatal notch, Passavant’s ridge, and aberrant pulsations.

Original languageEnglish (US)
Pages (from-to)720-728
Number of pages9
JournalOtolaryngology - Head and Neck Surgery (United States)
Volume160
Issue number4
DOIs
StatePublished - Apr 1 2019

Fingerprint

Soft Palate
Velopharyngeal Insufficiency
Nonparametric Statistics
Reproducibility of Results
Prospective Studies
Pediatrics

Keywords

  • VPI
  • interrater reliability
  • velopharyngeal insufficiency

ASJC Scopus subject areas

  • Surgery
  • Otorhinolaryngology

Cite this

Miller, Craig ; Bly, Randall ; Cofer, Shelagh ; Dahl, John P. ; de Serres, Lianne ; Goudy, Steven ; Hartzell, Larry ; Jabbour, Noel ; Kacmarysnki, Deborah ; Macarthur, Carol ; Messner, Anna ; Milczuk, Henry ; Rastatter, Jeff C ; Sweibel Rosenthal, Laura ; Scott, Andrew ; Tollefson, Travis ; Woolley, Audie ; Zdanski, Carlton ; Zopf, Dave A. ; Sie, Kathleen. / Multicenter Interrater Reliability in the Endoscopic Assessment of Velopharyngeal Function Using a Video Instruction Tool. In: Otolaryngology - Head and Neck Surgery (United States). 2019 ; Vol. 160, No. 4. pp. 720-728.
@article{96b6f943173a41baa8ed5bfa6863520c,
title = "Multicenter Interrater Reliability in the Endoscopic Assessment of Velopharyngeal Function Using a Video Instruction Tool",
abstract = "Objective: Assess interrater agreement of endoscopic assessment of velopharyngeal (VP) function before and after viewing the video instruction tool (VIT). We hypothesized improvement in interrater agreement using the Golding-Kushner scale (GKS) after viewing the VIT. Study Design: Prospective study. Setting: Multi-institutional. Methods: Sixteen fellowship-trained pediatric otolaryngologists who treat velopharyngeal insufficiency (VPI) rated 50 video segments using the GKS before and after watching the VIT. Raters assessed gap size percentage and lateral pharyngeal wall (LPW), soft palate (SP), and posterior pharyngeal wall (PPW) movement. Intraclass correlation coefficient was estimated for these continuous measures. Raters also indicated the presence of a palatal notch, Passavant’s ridge, and aberrant pulsations (categorical variables). Fleiss κ coefficient was used for categorical variables. Wilcoxon signed-rank test was performed on the difference between the pre/post individual video ratings. Results: Reliability improved for all continuous variables after watching the instructional video. The improvement was significant for PPW (0.22-0.30, P <.001), SP (left: 0.63-0.68, P <.001 and right: 0.64-0.68, P =.001), and LPW (left: 0.49-0.54, P =.01 and right: 0.49-0.54, P =.09) but not significant for gap size (0.65-0.69, P =.36). Among categorical variables, agreement on Passavant’s ridge significantly improved (0.30-0.36, P =.03). Conclusion: Exposure to a video instruction tool improves interrater agreement of endoscopic assessment of VP function. Significant improvement was observed in our primary end points, specifically posterior pharyngeal wall movement, soft palate movement, and lateral pharyngeal wall movement. There was less impact of the VIT on the interrater agreement of the categorical variables, palatal notch, Passavant’s ridge, and aberrant pulsations.",
keywords = "VPI, interrater reliability, velopharyngeal insufficiency",
author = "Craig Miller and Randall Bly and Shelagh Cofer and Dahl, {John P.} and {de Serres}, Lianne and Steven Goudy and Larry Hartzell and Noel Jabbour and Deborah Kacmarysnki and Carol Macarthur and Anna Messner and Henry Milczuk and Rastatter, {Jeff C} and {Sweibel Rosenthal}, Laura and Andrew Scott and Travis Tollefson and Audie Woolley and Carlton Zdanski and Zopf, {Dave A.} and Kathleen Sie",
year = "2019",
month = "4",
day = "1",
doi = "10.1177/0194599818822989",
language = "English (US)",
volume = "160",
pages = "720--728",
journal = "Otolaryngology - Head and Neck Surgery",
issn = "0194-5998",
publisher = "Mosby Inc.",
number = "4",

}

Miller, C, Bly, R, Cofer, S, Dahl, JP, de Serres, L, Goudy, S, Hartzell, L, Jabbour, N, Kacmarysnki, D, Macarthur, C, Messner, A, Milczuk, H, Rastatter, JC, Sweibel Rosenthal, L, Scott, A, Tollefson, T, Woolley, A, Zdanski, C, Zopf, DA & Sie, K 2019, 'Multicenter Interrater Reliability in the Endoscopic Assessment of Velopharyngeal Function Using a Video Instruction Tool', Otolaryngology - Head and Neck Surgery (United States), vol. 160, no. 4, pp. 720-728. https://doi.org/10.1177/0194599818822989

Multicenter Interrater Reliability in the Endoscopic Assessment of Velopharyngeal Function Using a Video Instruction Tool. / Miller, Craig; Bly, Randall; Cofer, Shelagh; Dahl, John P.; de Serres, Lianne; Goudy, Steven; Hartzell, Larry; Jabbour, Noel; Kacmarysnki, Deborah; Macarthur, Carol; Messner, Anna; Milczuk, Henry; Rastatter, Jeff C; Sweibel Rosenthal, Laura; Scott, Andrew; Tollefson, Travis; Woolley, Audie; Zdanski, Carlton; Zopf, Dave A.; Sie, Kathleen.

In: Otolaryngology - Head and Neck Surgery (United States), Vol. 160, No. 4, 01.04.2019, p. 720-728.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Multicenter Interrater Reliability in the Endoscopic Assessment of Velopharyngeal Function Using a Video Instruction Tool

AU - Miller, Craig

AU - Bly, Randall

AU - Cofer, Shelagh

AU - Dahl, John P.

AU - de Serres, Lianne

AU - Goudy, Steven

AU - Hartzell, Larry

AU - Jabbour, Noel

AU - Kacmarysnki, Deborah

AU - Macarthur, Carol

AU - Messner, Anna

AU - Milczuk, Henry

AU - Rastatter, Jeff C

AU - Sweibel Rosenthal, Laura

AU - Scott, Andrew

AU - Tollefson, Travis

AU - Woolley, Audie

AU - Zdanski, Carlton

AU - Zopf, Dave A.

AU - Sie, Kathleen

PY - 2019/4/1

Y1 - 2019/4/1

N2 - Objective: Assess interrater agreement of endoscopic assessment of velopharyngeal (VP) function before and after viewing the video instruction tool (VIT). We hypothesized improvement in interrater agreement using the Golding-Kushner scale (GKS) after viewing the VIT. Study Design: Prospective study. Setting: Multi-institutional. Methods: Sixteen fellowship-trained pediatric otolaryngologists who treat velopharyngeal insufficiency (VPI) rated 50 video segments using the GKS before and after watching the VIT. Raters assessed gap size percentage and lateral pharyngeal wall (LPW), soft palate (SP), and posterior pharyngeal wall (PPW) movement. Intraclass correlation coefficient was estimated for these continuous measures. Raters also indicated the presence of a palatal notch, Passavant’s ridge, and aberrant pulsations (categorical variables). Fleiss κ coefficient was used for categorical variables. Wilcoxon signed-rank test was performed on the difference between the pre/post individual video ratings. Results: Reliability improved for all continuous variables after watching the instructional video. The improvement was significant for PPW (0.22-0.30, P <.001), SP (left: 0.63-0.68, P <.001 and right: 0.64-0.68, P =.001), and LPW (left: 0.49-0.54, P =.01 and right: 0.49-0.54, P =.09) but not significant for gap size (0.65-0.69, P =.36). Among categorical variables, agreement on Passavant’s ridge significantly improved (0.30-0.36, P =.03). Conclusion: Exposure to a video instruction tool improves interrater agreement of endoscopic assessment of VP function. Significant improvement was observed in our primary end points, specifically posterior pharyngeal wall movement, soft palate movement, and lateral pharyngeal wall movement. There was less impact of the VIT on the interrater agreement of the categorical variables, palatal notch, Passavant’s ridge, and aberrant pulsations.

AB - Objective: Assess interrater agreement of endoscopic assessment of velopharyngeal (VP) function before and after viewing the video instruction tool (VIT). We hypothesized improvement in interrater agreement using the Golding-Kushner scale (GKS) after viewing the VIT. Study Design: Prospective study. Setting: Multi-institutional. Methods: Sixteen fellowship-trained pediatric otolaryngologists who treat velopharyngeal insufficiency (VPI) rated 50 video segments using the GKS before and after watching the VIT. Raters assessed gap size percentage and lateral pharyngeal wall (LPW), soft palate (SP), and posterior pharyngeal wall (PPW) movement. Intraclass correlation coefficient was estimated for these continuous measures. Raters also indicated the presence of a palatal notch, Passavant’s ridge, and aberrant pulsations (categorical variables). Fleiss κ coefficient was used for categorical variables. Wilcoxon signed-rank test was performed on the difference between the pre/post individual video ratings. Results: Reliability improved for all continuous variables after watching the instructional video. The improvement was significant for PPW (0.22-0.30, P <.001), SP (left: 0.63-0.68, P <.001 and right: 0.64-0.68, P =.001), and LPW (left: 0.49-0.54, P =.01 and right: 0.49-0.54, P =.09) but not significant for gap size (0.65-0.69, P =.36). Among categorical variables, agreement on Passavant’s ridge significantly improved (0.30-0.36, P =.03). Conclusion: Exposure to a video instruction tool improves interrater agreement of endoscopic assessment of VP function. Significant improvement was observed in our primary end points, specifically posterior pharyngeal wall movement, soft palate movement, and lateral pharyngeal wall movement. There was less impact of the VIT on the interrater agreement of the categorical variables, palatal notch, Passavant’s ridge, and aberrant pulsations.

KW - VPI

KW - interrater reliability

KW - velopharyngeal insufficiency

UR - http://www.scopus.com/inward/record.url?scp=85060922359&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85060922359&partnerID=8YFLogxK

U2 - 10.1177/0194599818822989

DO - 10.1177/0194599818822989

M3 - Article

C2 - 30642231

AN - SCOPUS:85060922359

VL - 160

SP - 720

EP - 728

JO - Otolaryngology - Head and Neck Surgery

JF - Otolaryngology - Head and Neck Surgery

SN - 0194-5998

IS - 4

ER -