ChatGPT in radiology: A systematic review of performance, pitfalls, and future perspectives

Pedram Keshavarz, Sara Bagherieh, Seyed Ali Nabipoorashrafi, Hamid Chalian, Amir Ali Rahsepar, Grace Hyun J. Kim, Cameron Hassani, Steven S. Raman, Arash Bedayat*

*Corresponding author for this work

Research output: Contribution to journalReview articlepeer-review

1 Scopus citations

Abstract

Purpose: The purpose of this study was to systematically review the reported performances of ChatGPT, identify potential limitations, and explore future directions for its integration, optimization, and ethical considerations in radiology applications. Materials and methods: After a comprehensive review of PubMed, Web of Science, Embase, and Google Scholar databases, a cohort of published studies was identified up to January 1, 2024, utilizing ChatGPT for clinical radiology applications. Results: Out of 861 studies derived, 44 studies evaluated the performance of ChatGPT; among these, 37 (37/44; 84.1%) demonstrated high performance, and seven (7/44; 15.9%) indicated it had a lower performance in providing information on diagnosis and clinical decision support (6/44; 13.6%) and patient communication and educational content (1/44; 2.3%). Twenty-four (24/44; 54.5%) studies reported the proportion of ChatGPT's performance. Among these, 19 (19/24; 79.2%) studies recorded a median accuracy of 70.5%, and in five (5/24; 20.8%) studies, there was a median agreement of 83.6% between ChatGPT outcomes and reference standards [radiologists’ decision or guidelines], generally confirming ChatGPT's high accuracy in these studies. Eleven studies compared two recent ChatGPT versions, and in ten (10/11; 90.9%), ChatGPTv4 outperformed v3.5, showing notable enhancements in addressing higher-order thinking questions, better comprehension of radiology terms, and improved accuracy in describing images. Risks and concerns about using ChatGPT included biased responses, limited originality, and the potential for inaccurate information leading to misinformation, hallucinations, improper citations and fake references, cybersecurity vulnerabilities, and patient privacy risks. Conclusion: Although ChatGPT's effectiveness has been shown in 84.1% of radiology studies, there are still multiple pitfalls and limitations to address. It is too soon to confirm its complete proficiency and accuracy, and more extensive multicenter studies utilizing diverse datasets and pre-training techniques are required to verify ChatGPT's role in radiology.

Original languageEnglish (US)
Pages (from-to)251-265
Number of pages15
JournalDiagnostic and interventional imaging
Volume105
Issue number7-8
DOIs
StatePublished - Jul 1 2024
Externally publishedYes

Keywords

  • Artificial intelligence
  • ChatGPT
  • Decision support systems
  • Large language model
  • OpenAI

ASJC Scopus subject areas

  • Radiological and Ultrasound Technology
  • Radiology Nuclear Medicine and imaging

Fingerprint

Dive into the research topics of 'ChatGPT in radiology: A systematic review of performance, pitfalls, and future perspectives'. Together they form a unique fingerprint.

Cite this