Artificial intelligence algorithm improves radiologist performance in skeletal age assessment: A prospective multicenter randomized controlled trial

David K. Eng; Nishith B. Khandwala; Jin Long; Nancy R. Fefferman; Shailee V. Lala; Naomi A. Strubel; Sarah S. Milla; Ross W. Filice; Susan E. Sharp; Alexander J. Towbin; Michael L. Francavilla; Summer L. Kaplan; Kirsten Ecklund; Sanjay P. Prabhu; Brian J. Dillon; Brian M. Everist; Christopher G. Anton; Mark E. Bittman; Rebecca Dennis; David B. Larson; Jayne M. Seekins; Cicero T. Silva; Arash R. Zandieh; Curtis P. Langlotz; Matthew P. Lungren; Safwan S. Halabi

doi:10.1148/radiol.2021204021

Artificial intelligence algorithm improves radiologist performance in skeletal age assessment: A prospective multicenter randomized controlled trial

David K. Eng^*, Nishith B. Khandwala, Jin Long, Nancy R. Fefferman, Shailee V. Lala, Naomi A. Strubel, Sarah S. Milla, Ross W. Filice, Susan E. Sharp, Alexander J. Towbin, Michael L. Francavilla, Summer L. Kaplan, Kirsten Ecklund, Sanjay P. Prabhu, Brian J. Dillon, Brian M. Everist, Christopher G. Anton, Mark E. Bittman, Rebecca Dennis, David B. LarsonJayne M. Seekins, Cicero T. Silva, Arash R. Zandieh, Curtis P. Langlotz, Matthew P. Lungren, Safwan S. Halabi

^*Corresponding author for this work

Radiology

Research output: Contribution to journal › Article › peer-review

53 Scopus citations

Abstract

Background: Previous studies suggest that use of artificial intelligence (AI) algorithms as diagnostic aids may improve the quality of skeletal age assessment, though these studies lack evidence from clinical practice. Purpose: To compare the accuracy and interpretation time of skeletal age assessment on hand radiograph examinations with and without the use of an AI algorithm as a diagnostic aid. Materials and Methods: In this prospective randomized controlled trial, the accuracy of skeletal age assessment on hand radiograph examinations was performed with (n = 792) and without (n = 739) the AI algorithm as a diagnostic aid. For examinations with the AI algorithm, the radiologist was shown the AI interpretation as part of their routine clinical work and was permitted to accept or modify it. Hand radiographs were interpreted by 93 radiologists from six centers. The primary efficacy outcome was the mean absolute difference between the skeletal age dictated into the radiologists' signed report and the average interpretation of a panel of four radiologists not using a diagnostic aid. The secondary outcome was the interpretation time. A linear mixed-effects regression model with random center- and radiologist-level effects was used to compare the two experimental groups. Results: Overall mean absolute difference was lower when radiologists used the AI algorithm compared with when they did not (5.36 months vs 5.95 months; P = .04). The proportions at which the absolute difference exceeded 12 months (9.3% vs 13.0%, P = .02) and 24 months (0.5% vs 1.8%, P = .02) were lower with the AI algorithm than without it. Median radiologist interpretation time was lower with the AI algorithm than without it (102 seconds vs 142 seconds, P = .001). Conclusion: Use of an artificial intelligence algorithm improved skeletal age assessment accuracy and reduced interpretation times for radiologists, although differences were observed between centers.

Original language	English (US)
Pages (from-to)	692-699
Number of pages	8
Journal	Radiology
Volume	301
Issue number	3
DOIs	https://doi.org/10.1148/radiol.2021204021
State	Published - Dec 2021

Funding

Disclosures of Conflicts of Interest: D.K.E. disclosed no relevant relationships. N.B.K. is the cofounder of Bunkerhill Health. J.L. disclosed no relevant relationships. N.R.F. disclosed no relevant relationships. S.V.L. disclosed no relevant relationships. N.A.S. disclosed no relevant relationships. S.S.M. disclosed no relevant relationships. R.W.F. is on the Bunkerhill Health advisory board in exchange for a 0.5% stake in the company, received an honorarium from the Korean Society of Radiology for speaking at the Korean Congress of Radiology in 2019. S.E.S. disclosed no relevant relationships. A.J.T. is a consultant for Applied Radiology; institution received grants from Guerbet and the Cystic Fibrosis Foundation; receives royalties from Elsevier. M.L.F. disclosed no relevant relationships. S.L.K. disclosed no relevant relationships. K.E. disclosed no relevant relationships. S.P.P. disclosed no relevant relationships. B.J.D. disclosed no relevant relationships. B.M.E. disclosed no relevant relationships. C.G.A. receives book royalties from Amirsys. M.E.B. disclosed no relevant relationships. R.D. disclosed no relevant relationships. D.B.L. institution receives research support from Siemens Healthineers, holds stock in Bunkerhill Health. J.M.S. disclosed no relevant relationships. C.T.S. disclosed no relevant relationships. A.R.Z. disclosed no relevant relationships. C.P.L. received stock in Bunkerhill Health in return for board membership; received stock options in whiterabbit.ai, Nines.ai, GalileoCDS, and Sirona Medical for service on their advisory boards; institution received grants from Bunkerhill Health, Carestream, GE Healthcare, Google Cloud, IBM, IDEXX, Lambda, Lunit, Nines, Subtle Medical, whiterabbit.ai, Bayer, Fuji, and Kheiron; received honoraria for virtual presentations at radiology meetings from Canon and auntminnie.com; was reimbursed for travel by Siemens. M.P.L. is on the board of Carestream and Nines Radiology; is a consultant for Bayer and Microsoft; holds stock in Bunkerhill Health. S.S.H. disclosed no relevant relationships.

ASJC Scopus subject areas

Radiology Nuclear Medicine and imaging

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1148/radiol.2021204021

Cite this

Eng, D. K., Khandwala, N. B., Long, J., Fefferman, N. R., Lala, S. V., Strubel, N. A., Milla, S. S., Filice, R. W., Sharp, S. E., Towbin, A. J., Francavilla, M. L., Kaplan, S. L., Ecklund, K., Prabhu, S. P., Dillon, B. J., Everist, B. M., Anton, C. G., Bittman, M. E., Dennis, R., ... Halabi, S. S. (2021). Artificial intelligence algorithm improves radiologist performance in skeletal age assessment: A prospective multicenter randomized controlled trial. Radiology, 301(3), 692-699. https://doi.org/10.1148/radiol.2021204021

@article{d4326eaf00544b2d8eb3bfcc68536d7d,

title = "Artificial intelligence algorithm improves radiologist performance in skeletal age assessment: A prospective multicenter randomized controlled trial",

abstract = "Background: Previous studies suggest that use of artificial intelligence (AI) algorithms as diagnostic aids may improve the quality of skeletal age assessment, though these studies lack evidence from clinical practice. Purpose: To compare the accuracy and interpretation time of skeletal age assessment on hand radiograph examinations with and without the use of an AI algorithm as a diagnostic aid. Materials and Methods: In this prospective randomized controlled trial, the accuracy of skeletal age assessment on hand radiograph examinations was performed with (n = 792) and without (n = 739) the AI algorithm as a diagnostic aid. For examinations with the AI algorithm, the radiologist was shown the AI interpretation as part of their routine clinical work and was permitted to accept or modify it. Hand radiographs were interpreted by 93 radiologists from six centers. The primary efficacy outcome was the mean absolute difference between the skeletal age dictated into the radiologists' signed report and the average interpretation of a panel of four radiologists not using a diagnostic aid. The secondary outcome was the interpretation time. A linear mixed-effects regression model with random center- and radiologist-level effects was used to compare the two experimental groups. Results: Overall mean absolute difference was lower when radiologists used the AI algorithm compared with when they did not (5.36 months vs 5.95 months; P = .04). The proportions at which the absolute difference exceeded 12 months (9.3% vs 13.0%, P = .02) and 24 months (0.5% vs 1.8%, P = .02) were lower with the AI algorithm than without it. Median radiologist interpretation time was lower with the AI algorithm than without it (102 seconds vs 142 seconds, P = .001). Conclusion: Use of an artificial intelligence algorithm improved skeletal age assessment accuracy and reduced interpretation times for radiologists, although differences were observed between centers.",

author = "Eng, {David K.} and Khandwala, {Nishith B.} and Jin Long and Fefferman, {Nancy R.} and Lala, {Shailee V.} and Strubel, {Naomi A.} and Milla, {Sarah S.} and Filice, {Ross W.} and Sharp, {Susan E.} and Towbin, {Alexander J.} and Francavilla, {Michael L.} and Kaplan, {Summer L.} and Kirsten Ecklund and Prabhu, {Sanjay P.} and Dillon, {Brian J.} and Everist, {Brian M.} and Anton, {Christopher G.} and Bittman, {Mark E.} and Rebecca Dennis and Larson, {David B.} and Seekins, {Jayne M.} and Silva, {Cicero T.} and Zandieh, {Arash R.} and Langlotz, {Curtis P.} and Lungren, {Matthew P.} and Halabi, {Safwan S.}",

year = "2021",

month = dec,

doi = "10.1148/radiol.2021204021",

language = "English (US)",

volume = "301",

pages = "692--699",

journal = "Radiology",

issn = "0033-8419",

publisher = "Radiological Society of North America Inc.",

number = "3",

}

Eng, DK, Khandwala, NB, Long, J, Fefferman, NR, Lala, SV, Strubel, NA, Milla, SS, Filice, RW, Sharp, SE, Towbin, AJ, Francavilla, ML, Kaplan, SL, Ecklund, K, Prabhu, SP, Dillon, BJ, Everist, BM, Anton, CG, Bittman, ME, Dennis, R, Larson, DB, Seekins, JM, Silva, CT, Zandieh, AR, Langlotz, CP, Lungren, MP & Halabi, SS 2021, 'Artificial intelligence algorithm improves radiologist performance in skeletal age assessment: A prospective multicenter randomized controlled trial', Radiology, vol. 301, no. 3, pp. 692-699. https://doi.org/10.1148/radiol.2021204021

TY - JOUR

T1 - Artificial intelligence algorithm improves radiologist performance in skeletal age assessment

T2 - A prospective multicenter randomized controlled trial

AU - Eng, David K.

AU - Khandwala, Nishith B.

AU - Long, Jin

AU - Fefferman, Nancy R.

AU - Lala, Shailee V.

AU - Strubel, Naomi A.

AU - Milla, Sarah S.

AU - Filice, Ross W.

AU - Sharp, Susan E.

AU - Towbin, Alexander J.

AU - Francavilla, Michael L.

AU - Kaplan, Summer L.

AU - Ecklund, Kirsten

AU - Prabhu, Sanjay P.

AU - Dillon, Brian J.

AU - Everist, Brian M.

AU - Anton, Christopher G.

AU - Bittman, Mark E.

AU - Dennis, Rebecca

AU - Larson, David B.

AU - Seekins, Jayne M.

AU - Silva, Cicero T.

AU - Zandieh, Arash R.

AU - Langlotz, Curtis P.

AU - Lungren, Matthew P.

AU - Halabi, Safwan S.

PY - 2021/12

Y1 - 2021/12

N2 - Background: Previous studies suggest that use of artificial intelligence (AI) algorithms as diagnostic aids may improve the quality of skeletal age assessment, though these studies lack evidence from clinical practice. Purpose: To compare the accuracy and interpretation time of skeletal age assessment on hand radiograph examinations with and without the use of an AI algorithm as a diagnostic aid. Materials and Methods: In this prospective randomized controlled trial, the accuracy of skeletal age assessment on hand radiograph examinations was performed with (n = 792) and without (n = 739) the AI algorithm as a diagnostic aid. For examinations with the AI algorithm, the radiologist was shown the AI interpretation as part of their routine clinical work and was permitted to accept or modify it. Hand radiographs were interpreted by 93 radiologists from six centers. The primary efficacy outcome was the mean absolute difference between the skeletal age dictated into the radiologists' signed report and the average interpretation of a panel of four radiologists not using a diagnostic aid. The secondary outcome was the interpretation time. A linear mixed-effects regression model with random center- and radiologist-level effects was used to compare the two experimental groups. Results: Overall mean absolute difference was lower when radiologists used the AI algorithm compared with when they did not (5.36 months vs 5.95 months; P = .04). The proportions at which the absolute difference exceeded 12 months (9.3% vs 13.0%, P = .02) and 24 months (0.5% vs 1.8%, P = .02) were lower with the AI algorithm than without it. Median radiologist interpretation time was lower with the AI algorithm than without it (102 seconds vs 142 seconds, P = .001). Conclusion: Use of an artificial intelligence algorithm improved skeletal age assessment accuracy and reduced interpretation times for radiologists, although differences were observed between centers.

AB - Background: Previous studies suggest that use of artificial intelligence (AI) algorithms as diagnostic aids may improve the quality of skeletal age assessment, though these studies lack evidence from clinical practice. Purpose: To compare the accuracy and interpretation time of skeletal age assessment on hand radiograph examinations with and without the use of an AI algorithm as a diagnostic aid. Materials and Methods: In this prospective randomized controlled trial, the accuracy of skeletal age assessment on hand radiograph examinations was performed with (n = 792) and without (n = 739) the AI algorithm as a diagnostic aid. For examinations with the AI algorithm, the radiologist was shown the AI interpretation as part of their routine clinical work and was permitted to accept or modify it. Hand radiographs were interpreted by 93 radiologists from six centers. The primary efficacy outcome was the mean absolute difference between the skeletal age dictated into the radiologists' signed report and the average interpretation of a panel of four radiologists not using a diagnostic aid. The secondary outcome was the interpretation time. A linear mixed-effects regression model with random center- and radiologist-level effects was used to compare the two experimental groups. Results: Overall mean absolute difference was lower when radiologists used the AI algorithm compared with when they did not (5.36 months vs 5.95 months; P = .04). The proportions at which the absolute difference exceeded 12 months (9.3% vs 13.0%, P = .02) and 24 months (0.5% vs 1.8%, P = .02) were lower with the AI algorithm than without it. Median radiologist interpretation time was lower with the AI algorithm than without it (102 seconds vs 142 seconds, P = .001). Conclusion: Use of an artificial intelligence algorithm improved skeletal age assessment accuracy and reduced interpretation times for radiologists, although differences were observed between centers.

UR - http://www.scopus.com/inward/record.url?scp=85120054469&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85120054469&partnerID=8YFLogxK

U2 - 10.1148/radiol.2021204021

DO - 10.1148/radiol.2021204021

M3 - Article

C2 - 34581608

AN - SCOPUS:85120054469

SN - 0033-8419

VL - 301

SP - 692

EP - 699

JO - Radiology

JF - Radiology

IS - 3

ER -

Artificial intelligence algorithm improves radiologist performance in skeletal age assessment: A prospective multicenter randomized controlled trial

Abstract

Funding

ASJC Scopus subject areas

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this