Experimental design and data analysis in receiver operating characteristic studies: Lessons learned from reports in Radiology from 1997 to 2006

Junji Shiraishi*, Lorenzo L. Pesce, Charles E. Metz, Kunio Doi

*Corresponding author for this work

Research output: Contribution to journalArticle

42 Scopus citations

Abstract

Purpose: To provide a broad perspective concerning the recent use of receiver operating characteristic (ROC) analysis in medical imaging by reviewing ROC studies published in Radiology between 1997 and 2006 for experimental design, imaging modality, medical condition, and ROC paradigm. Materials and Methods: Two hundred ninety-five studies were obtained by conducting a literature search with PubMed with two criteria: publication in Radiology between 1997 and 2006 and occurrence of the phrase "receiver operating characteristic." Studies returned by the query that were not diagnostic imaging procedure performance evaluations were excluded. Characteristics of the remaining studies were tabulated. Results: Two hundred thirty-three (79.0%) of the 295 studies reported findings based on observers' diagnostic judgments or objective measurements. Forty-three (14.6%) did not include human observers, with most of these reporting an evaluation of a computer-aided diagnosis system or functional data obtained with computed tomography (CT) or magnetic resonance (MR) imaging. The remaining 19 (6.4%) studies were classified as reviews or meta-analyses and were excluded from our subsequent analysis. Among the various imaging modalities, MR imaging (46.0%) and CT (25.7%) were investigated most frequently. Approximately 60% (144 of 233) of ROC studies with human observers published in Radiology included three or fewer observers. Conclusion: ROC analysis is widely used in radiologic research, confirming its fundamental role in assessing diagnostic performance. However, the ROC studies reported in Radiology were not always adequate to support clear and clinically relevant conclusions.

Original languageEnglish (US)
Pages (from-to)822-830
Number of pages9
JournalRadiology
Volume253
Issue number3
DOIs
StatePublished - Dec 2009

ASJC Scopus subject areas

  • Radiology Nuclear Medicine and imaging

Fingerprint Dive into the research topics of 'Experimental design and data analysis in receiver operating characteristic studies: Lessons learned from reports in Radiology from 1997 to 2006'. Together they form a unique fingerprint.

  • Cite this