The Lung Image Database Consortium (LIDC): An Evaluation of Radiologist Variability in the Identification of Lung Nodules on CT Scans

Rationale and Objectives: The purpose of this study was to analyze the variability of experienced thoracic radiologists in the identification of lung nodules on computed tomography (CT) scans and thereby to investigate variability in the establishment of the "truth" against which nodule-based studies are measured. Materials and Methods: Thirty CT scans were reviewed twice by four thoracic radiologists through a two-phase image annotation process. During the initial "blinded read" phase, radiologists independently marked lesions they identified as "nodule ≥3 mm (diameter)," "nodule <3 mm," or "non-nodule ≥3 mm." During the subsequent "unblinded read" phase, the blinded read results of all four radiologists were revealed to each radiologist, who then independently reviewed their marks along with the anonymous marks of their colleagues; a radiologist's own marks then could be deleted, added, or left unchanged. This approach was developed to identify, as completely as possible, all nodules in a scan without requiring forced consensus. Results: After the initial blinded read phase, 71 lesions received "nodule ≥3 mm" marks from at least one radiologist; however, all four radiologists assigned such marks to only 24 (33.8%) of these lesions. After the unblinded reads, a total of 59 lesions were marked as "nodule ≥3 mm" by at least one radiologist. Twenty-seven (45.8%) of these lesions received such marks from all four radiologists, three (5.1%) were identified as such by three radiologists, 12 (20.3%) were identified by two radiologists, and 17 (28.8%) were identified by only a single radiologist. Conclusion: The two-phase image annotation process yields improved agreement among radiologists in the interpretation of nodules ≥3 mm. Nevertheless, substantial variabilty remains across radiologists in the task of lung nodule identification.